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PREFACE 


This  document  Is  the  result  of  a  collaborative  effort  of  the  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  the 
American  Psychological  Association  (APA),  and  the  Federation  of  Behavioral, 
Psychological  and  Cognitive  Sciences  (FBPCS). 

The  APA,  with  the  cooperation  of  FBPCS,  organized  a  workshop  on 
transfer  of  training  Issues  as  the  first  of  a  series  of  Department  of 
Defense  Research  Roundtable  meetings.  ARI  provided  travel  funds  for  the 
presenters  and  support,  through  Its  Scientific  Services  Program,  for  the 
preparation  of  this  summary  document. 

The  idea  for  the  Research  Roundtable  series  was  generated  by  the  APA 
Committee  on  Research  Support  (CORS),  a  committee  reporting  to  APA's  major 
scientific  policymaking  body,  the  Board  of  Scientific  Affairs.  CORS 
Initiated  the  project  to  provide  a  forum  for  the  discussion  of  behavioral 
research  Issues  related  to  the  defense  mission.  Transfer  of  training  was 
chosen  as  the  first  topic  for  consideration. 

CORS  set  the  framework  for  the  Roundtable  project,  and  appointed 
members  William  Howell  and  Gary  M.  Olson  to  provide  oversight. 

Subsequently,  a  planning  committee  developed  specific  guidelines  for  the 
workshop  and  selected  presenters.  Planning  Committee  members  were  Gary  M. 
Olson,  Irv  Goldstein,  David  Kleras,  and  William  Montague.  Throughout  the 
process.  Captain  Paul  Chateller,  Assistant  for  Training  and  Personnel 
Technology,  Office  of  the  Deputy  Assistant  Secretary  for  Research  and 
Technology,  and  Milton  Katz,  ARI  Director  of  Basic  Research,  provided 
guidance  and  suggestions. 

The  Individuals  selected  as  presenters  represent  a  spectrum  of 
Interests  and  experience  In  the  area  of  transfer  of  training,  including 
those  who  conduct  research  In  academic  settings,  those  who  conduct  research 
In  Department  of  Defense  laboratories,  and  those  who  both  conduct  research 
and  apply  findings  In  non-academic  settings.  Further,  the  planning 
committee  Invited  a  number  of  participants  to  comment  on  the  presentations, 
these  Individuals  represented  both  military  and  non-mil Itary  organizations. 
A  complete  listing  of  workshop  participants.  Including  those  who  made 
presentations,  may  be  found  on  page  Iv. 


Presenter*  were  guided  by  a  eet  of  Questions  prepared  by  the  planning 
committee  for  the  workshop;  those  questions  are  provided  on  page  vl.  All 
presenters  were  provided  with  transcriptions  of  their  remarks;  the 
Individual  papers  are  the  edited  versions  of  the  transcriptions.  One 
presenter,  Joe  Yasutake  of  the  Air  Force  Human  Resources  Laboratory  at  Lowry 
Air  Force  Base,  dig  not  prepare  a  paper  for  this  document. 

The  summary  and  recommendations  were  prepared  from  concluding  remarks 
made  by  all  participants. 


Special  thanks  are  due  to  Gary  M.  Olson,  who  chaired  the  workshop,  and 
to  Cynthia  H.  Null.  Executive  Director  of  FBPCS,  who  provided  much  support 
In  both  the  organizational  and  editorial  tasks  of  the  first  Roundtable 
aieet  Ing. 

Virginia  E.  Holt 

Office  of  Scientific  Affairs 

American  Psychological  Association 
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OoD  Research  Roundtable 
Transfer  of  Training 
February  27-28,  1986 

Guiding  Questions  for  Presenters 


1.  From  your  particular  specialization,  what  can  we  look  for  in 
the  future  in  the  area  of  transfer  of  training?  What 
approaches  do  you  expect  slight  be  developed?  With  what 
results? 

2.  What  should  be  taught  (e.g.,  skills?  strategies?)?  How  do  we 
find  out  what  should  be  taught?  How  should  the  amount  of 
transfer  of  knowledge  be  sieasured? 

3.  How  siuch  should  be  taught,  and  how  (e.g.,  classroom 
sltuat ion?)? 

4.  How  can  we  ensure/measure  transfer  of  training  (through 
performance)  on  the  Job? 

5.  Is  training  enough?  Is  it  worthwhile  to  train? 

6.  When  do  we  train  for  certain  skills?  What  skills  should  be 
emphasized  In  an  Initial  training  period?  What  skills  should 
be  Included  In  later /advanced  training  periods? 

7.  What  provisions  should  be  made  for  feedback  from  the  Job  to 
the  training  program? 

8.  How  does  personnel  selection  Influence  training  programs? 

9.  How  can  the  appropriate  training  programs  be  Implemented? 
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Will  lam  Johnson,  Ph.D. 


Search  Technology,  Inc. 


My  Intention  today  la  to  apeak  about  the  reality  of  transfer  of 
training  research.  The  presentation  will  be  based  on  sty  experiences  while 
conducting  transfer  of  training  assessment  In  technical  training 
environments.  I  will  first  describe  our  transfer  of  training  research  then 
will  respond  to  the  fist  of  questions  prepared  by  the  APA  and  suggested  as 
our  presentation  content. 

My  background  qualifies  me  to  discuss  the  reality  of  transfer  of 
training  assessment  In  technical  environments,  as  I  have  a  good  deal  of 
practical  experience  as  an  Instructor  and  ss  a  student  of  technical 
Instruction.  I  taught  In  technical  training  environments  Including  high 
school,  community  college,  and  university  for  over  ten  years.  I've  been 
Involved  In  a  number  of  technical  classes  In  the  Air  Force,  Navy,  Army,  and 
Industry  where  I've  participated  as  a  developer  as  well  as  a  receiver  (l.e., 
student)  of  instruction.  In  addition  I  have  formally  assessed  transfer  of 
training  In  a  number  of  technical  environments. 

My  experience  In  transfer  of  training  (since  1977)  has  been  in 
examining  how  a  person  transfers  from  simulation  oriented  computer-based 
simulation  to  real  equipment  dlagnost ic/troubleshoot Ing  performance.  It's 
from  that  dual  perspective  —  technical  Instructor  and  researcher  —  that  I 
approach  these  problems  and  make  this  presentation. 

When  considering  a  transfer  of  training  experiment,  one  must  think 
about  the  Importance  of  good  planning,  good  communications,  and  the 
Importance  of  coping  with  problems  that  might  arise  In  the  various  stages  of 
a  transfer  of  training  experimental  design. 

It  Is  difficult  to  plan  for  all  possibilities.  One  example  Is  an 
experiment  associated  with  my  dissertation.  The  transfer  task  required 
troubleshooting  operational  aircraft  engines  running  on  an  outdoor  test  pad. 
The  day  before  beginning  the  experiment,  I  came  down  with  pneumonia,  and  for 
the  rest  of  the  week  received  dally  physician  care  before  collecting  my 
data. 

In  that  same  experiment,  we  pilot  tested  our  Instruments  first  by 
collecting  the  troubleshooting  performance  of  a  number  of  example  subjects. 
We  assessed  the  pilot  test  data  to  be  sure  that  our  data  collection 
Instruments  were  appropriate.  We  were  reasonably  sure  that  the  Instruments 
would  permit  us  to  record  end  classify  all  of  the  actions  of  the  subjects. 
Our  best  laid  plans  met  with  difficulty  by  the  second  or  third  of  about  40 
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subjects.  With  that  particular  failure  the  aircraft  engine  (transfer  task) 
ran  out  of  gaa  aa  planned.  The  eubject  didn't  have  the  eilQhteat  Idea  what 
to  do,  therefore,  put  down  the  toolbox  and  cried.  I  didn't  have  anywhere  on 
aiy  fore  to  check  that  the  eubject  cried.  In  eplte  of  the  moat  detailed 
planning,  It  la  laposalble  to  anticipate  everything) 

My  example  of  the  Importance  of  clear  communlcat Ion  atoms  from  a  recent 
experiment  In  which  we  trained  a  group  of  unllcenaed  nuclear  power  plant 
operators.  He  used  a  computer  simulation  for  training  troubleshooting  of  an 
emergency  diesel  generator.  The  diesel  generator  la  a  standby  electric 
power  unit  for  a  nuclear  plant.  Me  transferred  people  from  either 
traditional  classroom  Instruction  (control  group)  or  the  computer  simulation 
(experimental  group)  Into  the  plant. 

Of  course,  the  person  collecting  the  data  needs  to  be  blind  to  the 
treatment.  Therefore,  the  classroom  Instructors  told  the  students, 

•Whatever  you  do,  don’t  tell  that  person  In  the  plant  what  you  did  here  In 
the  training  center.”  I  was  the  person  collecting  the  data.  The  first 
thing  I  said  to  each  eubject  was:  "It's  Important  that  you  do  not  tell  me 
what  you  did  In  the  training  center."  The  second  eubject  Immediately 
responded,  "I  won't  tell  you  what  I  did,  but  I'll  sure  tell  you  what  I 
didn't  do  and  that  was  the  computer  simulation."  Clearly  this  was  a 
communication  problem,  therefore  we  quickly  modified  the  Instructions. 

While  planning  and  communication  are  critically  Important  attributes 
for  effective  transfer  of  training  experiments  It  Is  also  necessary  for  the 
data  collection  personnel  to  be  adaptable  to  unexpected  situations.  The 
first  example  of  the  Importance  of  coping  skills  can  aleo  be  highlighted  by 
that  situation  where  the  technical  troubleshooting  eubject  cried  rather  than 
employing  tools  and  test  equipment.  A  second  example  of  researcher 
adaptability  Is  highlighted  by  two  extreme  examples  of  the  data  collection 
physical  environment  In  which  we  have  worked.  We  have  taken  data  while 
working  behind  the  cold  blowing  air  of  an  airplane  propeller  In  an  Illinois 
winter.  At  the  other  extreme  we  have  collected  data  In  the  nuclear  power 
plant  diesel  room  where  the  temperature  was  over  110  degrees  F.  When 
working  In  these  environments  we  remind  ourselves  of  the  expression  from 
graduate  school  that  "data  are  sacred!" 

I  hope  these  brief  stories  serve  to  emphasize  the  Importance  of 
planning  and  communications  as  well  as  the  need  for  the  transfer  of  training 
researcher  to  have  good  coping  skills  for  data  collection.  These  attributes 
•re  particularly  true  when  the  transfer  task  Involves  "real-world"  job 
performance.  I  would  now  like  to  briefly  characterize  our  major  transfer  of 
training  experiments.  The  moat  thorough  review  publications  of  this  work 
can  be  found  In  Rouse  and  Hunt  (1084)  and  Johnson  (1087). 

Table  1  characterizes  our  transfer  of  training  experiments  since  1977. 
We  started  with  a  computer  simulation  called  TASK  (troubleshooting  by 
Application  of  Structural  Knowledge)  that  was  context  free  (CF).  Subjects 
were  instructed  to  find  failures  within  an  abstract  network.  We  varied  such 
factors  as  the  aiding  In  training,  the  nature  of  the  feedback  received,  the 
number  of  problems  received,  and  the  complexity  of  the  problems.  In  the 
next  eet  of  experiments,  we  used  the  same  context  free  elmulation  and 
transferred  people  to  a  context  specific  (CS)  simulation,  name  FAULT 
(Eramework  for  Aiding  the  Understanding  of  Logical  Iroubleshoot Ing).  We 


TABLE  1:  TRANSFER  EXPERIMENTS  FOR  STUDYING  COMPUTER 
SIMULATIONS  FOR  DIAGNOSTIC  TRAINING 


Training 

Transfer 

Years 

Context  Free  (CF) 

to 

Context  Free 

77-78 

Context  Free  (CF) 

to 

Context  Specific  (CS) 

79 

CF&CS 

to 

Aircraft  powerplants 

80-81 

CS  St  real  equipment 

to 

Army  communications  equipment 

82-83 

CS  St  real  equipment 

to 

Marine  propulsion  systems 

S2&85 

CS  Sl  real  equipment 

to 

Nuclear  plant  safety  systems 

83-86 

)TL-  r.  .1*  •>  u  v».'A  yv  i\  i .Mvw.^r. 


worked  Initially  In  the  auto  mechanics  and  the  aviation  training 
environments.  Aa  Table  1  shows,  we  later  worked  In  the  applied  settings  of 
Army  Communications,  Marine  propulsion  systems,  end  finally  nuclear  power 
plant  safety  systems. 

One  of  our  earliest  findings  was  that  problem  solving  can  be  taught 
with  a  context  free  simulation.  That  led  us  to  become  Interested  In  how  the 
learner  might  transfer  that  knowledge  Into  real  world  problem  solving.  I 
will  say  more  about  these  findings  later  In  this  presentation. 

We  also  measured  the  training  effect  of  Job  aide  and  ongoing  feedback 
on  post-training  performance.  We  used  a  Job  aid  to  provide  on-line 
"bookkeeping*  to  help  the  learner  keep  track  of  Information  during 
troubleshooting.  The  subjects  trained  with  the  on-line  bookkeeping  did 
better  In  the  transfer  task  than  groups  trained  without  that  aid. 

We  also  studied  the  effects  of  various  feedback  during  on-line 
diagnostic  training.  We  found  that  feedback  should  sxptaln  when  the  person 
Is  making  mistakes  --  not  Just  typical  kinds  of  feedback  provided  by  some 
computer  based  Instruction  that  make  wlerd  nolees  when  an  error  Is 
committed.  An  example  of  the  kind  of  feedback  we  provided  Is:  "That  Is  an 
Incorrect  and  unnecessary  step  because  your  previous  actions  have  already 
provided  you  with  that  Information." 

We  found  that  a  context  specific  focus  was  necessary  to  do  relevant 
transfer  assessment  for  "real  world"  troubleshooting  Job  skills.  I  believe 
that  if  we  train  a  person  to  do  a  Job  (e.g.,  troubleshoot  a  piece  of 
technical  equipment)  the  transfer  research  must  look  at  the  success  of  the 
learner  transferring  from  the  training  environment  to  the  real  equipment. 

We  found  that  skilled  technicians  more  readily  benefit  from  generic 
diagnostic  training  than  do  novice  technicians.  The  skilled  technicians  who 
were  asked  to  troubleshoot  an  abstract  network  related  the  task  to  their 
real  world  troubleshooting  tasks.  The  novice  technicians  said,  "What's 
this?  How  does  this  help  me  learn  to  troubleshoot?"  Consequently,  the 
novices  made  more  mistakes  In  the  context  free  system.  This  finding 
suggests  that  novice  trainees  should  initial fy  be  given  simulation  with  a 
reasonably  high  level  of  physical  fidelity. 

In  another  example,  students  showed  a  preference  for  a  rich  feedback 
over  "bells  and  whistles."  As  that  line  of  research  has  evolved,  the 
feedback  has  become  far  more  sophisticated,  based  on  the  specific  mistakes 
made  by  learners.  As  we  keep  raising  the  fidelity  of  the  simulation,  we 
have  reached  a  point  where  the  studente  are  more  attracted  to  the  part  of 
the  simulation  that  provides  rich  feedback,  rather  than  the  higher  fidelity 
part  of  the  simulation.  The  bells  and  whistles  approach  hlgh-f Idel Ity,  and 
In  my  opinion  that's  not  necessarily  whers  we  need  to  go  with  diagnostic 
training  simulations.  Our  research  has  emphasized  the  Importance  of 
cognitive  fidelity  over  physical  fidelity.  A  technical  Instructor  st  the 
Army  Signal  School  at  Fort  Cordon  told  me  that  our  simulations  were  "heads- 
on"  rather  than  "hands-on." 

During  the  remainder  of  my  allotted  time,  l  will  answer  the  questions 
that  were  given  to  us  as  a  guideline  for  our  presentations. 
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Why  do  a  transfer  of  training  design?  My  response  to  that  question  is 
that  transfer  of  training  experimentation  Is  a  reliable  means  to  assess  the 
present  training.  It's  a  good  method  of  comparing  training  siethods,  and 
It's  also  good  for  new  device  evaluation.  If  Job  training  Is  the  goal,  It 
seems  that  transfer  Is  a  way  of  measuring  whether  you've  accomplished  that 
goal . 

Unfortunately,  transfer  of  training  assessment  Is  expensive,  especially 
If  on-the-job  performance  Is  measured.  Planning,  data  collection  and 
analysis,  and  report  writing  all  require  personnel  —  researchers  and 
technical  experts  who  must  be  paid.  One  potential  solution  Is  to  allow  or 
encourage  the  researcher  to  become  something  of  a  technical  expert.  It  has 
saved  a  great  deal  of  time  In  our  research,  even  with  equipment  systems  as 
diverse  as  telephone  switchboards,  diesel  generators,  or  helicopter  blade 
systems. 

I  think  It's  Important  to  the  quality  of  our  research  that  we 
understand  the  technical  environment,  and  I  realize  It's  often  difficult  to 
do  that.  For  example.  If  becoming  a  technical  expert  takes  two  years.  It  Is 
not  practical  for  the  researcher  to  also  become  a  technical  expert.  The 
team,  however,  must  strive  to  obtain  at  least  an  ability  to  use  and 
understand  some  of  the  Jargon  of  the  particular  technical  domain. 

Another  answer  to  why  not  do  a  transfer  assessment  when  the 
organization,  after  seeing  that  the  experimental  treatment  looks  good, 
decides  to  train  everyone  with  only  the  experimental  treatment. 

Consequently,  there  Is  no  control  group  for  the  traditional  training,  making 
It  Impossible  to  do  a  proper  comparison.  While  such  a  situation  does  not 
preclude  a  transfer  aieasure,  It  does  eliminate  the  baseline  measure  provided 
by  a  control  group. 

Another  problem  is  that  training  transfer  experiments  are  sometimes 
Just  not  required  by  management.  They  are  not  necessarily  needed  to  see 
that  transfer  has  occurred,  so  It  saves  a  lot  of  time  and  money  not  to  do 
It.  Industrial  training  Is  usually  chartered  to  train  rather  than  do 
training  research. 


What  are  some  of  the  logistics  Involved  In  a  transfer  of  training 
experiment?  The  first  task  Is  to  convince  the  organization  of  the  value  of 
the  research.  The  plan  needs  to  be  translated  Into  some  language  that  makes 
sense  to  the  people  running  the  training  organization.  Potential  cost 
savings,  for  example.  Is  very  motivational  to  training  managers.  The  second 
task  Is  to  prepare  a  workable  experimental  design  for  the  particular 
operations  environment.  You  then  design  the  exper I mental  treatments, 
deliver  the  training,  and  collect  and  analyze  the  data.  Although  that  may 
seem  simple,  so  much  can  go  wrong!  Equipment  becomes  unavailable, 
Inetructlonal  staff  changes,  and  subjects  become  unavailable  or 
"contaminated"  by  experience. 

I  think  It's  Important  to  get  the  managers  and  Instructors  Involved  and 
psychologically  "Invsatsd"  In  the  project.  However,  Interestingly, 
sometimes  you  can  have  too  much  organlzat lonal  support.  If  everyone  becomes 
too  Interested  In  the  experimental  training,  like  a  computer  simulation. 

It's  pretty  rough  to  run  the  control  treatment  as  previously  mentioned. 
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Two  Of  the  APA  questions,  *2  and  *7,  *ddr****d  topic*  r*lat*d  to 
teaching  skill  or  strategy.  If  you  plan  to  hav*  strategy-based  Inatruction, 
It  I*  Important  to  have  some  contextual  focua.  Just  giving  a  paraon  generic 
rule*  for  troubleshooting  doaan't  work  nearly  aa  wall  aa  providing  the 
general  rule*  related  to  the  real  equipment  familiar  to  the  learner. 

Further,  the  atudent  need*  a  reason  to  learn.  The  student  aiuat 
perceive  that  the  training  Is  likely  to  reduce  the  uncertainty  associated 
with  on-the-job  troubleshooting.  Thay  need  to  eee  that  practicing  In  the 
training  environment  le  going  to  somehow  help  them  when  they  get  out  Into 
the  field.  I  think  we  need  to  build  confidence  through  etructured  practice. 

What  do  I  eee  as  the  future  of  transfer  assessment?  i  am  Interested  In 
looking  at  the  longevity  of  simulation  devices  —  why  one  device  seems  to 
stay  In  the  training  program  while  others  are  used  only  as  a  research  tool 
and  then  disappear.  What  are  the  Implementation  methods  of  certain  new 
training,  while  one  kind  of  training  stays  and  another  kind  of  training 
dies?  While  I  do  not  Intend  to  answer  the  question  here,  I  must  hint  that 
part  of  the  answer  la  found  In  how  technical  expert*  are  Involved  In 
conceptual Izat Ion  design,  Implementation,  and  evaluation  Instruction. 

Two  additional  areas  that  could  affect  transfer  of  training  research 
are  related  to  embedded  training  and  expert  systems.  With  embedded 
training,  we  ought  to  be  able  to  collect  data  from  the  person  while  they're 
on  the  Job  without  having  a  researcher  there  watching  every  move  they  make. 
The  attention  to  student  modeling  In  expert  system/ Intel  I Igent  tutoring 
systems  will  also  serve  to  enhance  our  understanding  of  learning  transfer. 

It  seems  to  me  that  these  areas  will  make  substantive  contributions  to  the 
future  of  transfer  of  training  research. 
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I  would  like  to  discuss  the  application  of  transfer  of  training 
research.  I'd  like  to  focus  on  DoD'a  UANPRIMT  program,  because  I  think  It 
poses  some  general  problems  In  the  application  of  transfer  research  and 
other  human  performance  data. 

manprint  le  essentially  a  eat  of  procedures  and  guidelines  formulated 
to  Insure  that  the  design  of  weapon  aystems  and  other  complex  hardware  and 
software  systems  would  be  compatible  with  personnel  and  training  resources. 

In  order  to  achieve  transfer  from  training  systems  to  weapon  systems. 

It  Is  essential  to  specify  the  human  performance  characteristics  of 
projected  eystems.  This  goes  beyond  the  usual  consideration  of  human 
factors.  Important  Information  Includes:  the  level  of  difficulty,  the  type 
of  training  to  be  administered,  the  minimum  aptitude  levels,  and  the 
approximate  length  of  training  needed  to  obtain  satisfactory  performance. 

In  order  to  be  effective  In  guiding  eystems  design,  these  types  of 
training  or  performance  analyses  need  to  be  provided  while  the  system  Is  on 
the  drawing  board. 

The  challenge  for  those  of  us  who  provide  training  design  is  to  develop 
precise  estimates  of  the  training  or  performance  Implications  of  prototype 
or  developmental  systems. 

The  flip  side  of  that  challenge  Is  the  opportunity  to  have  a  major  say 
at  a  point  In  the  system  when  It's  still  possible  to  make  a  difference. 
Typically,  we  are  presented  with  a  fait  aecomplie  —  the  system  Is  already 
In  production  or  operation,  and  we  are  asked  to  fix  It. 

Occasionally  It  Is  posslbls  to  fix  a  problem  here  or  there  In  such 
situations,  but  often  there  are  deeper  problems.  Of  course,  it  Is  unlikely 
that  an  unworkable  million  dollar  training  system  will  be  scrapped  — 
particularly  on  the  advice  of  a  training  psychologist. 

There  are  two  types  of  situations  In  the  transfer  world:  the  transfer 
of  learning,  when  we're  not  quite  sure  what  the  transfer  task  is,  and  the 
transfer  of  training,  where  the  target  task  Is  generally  known  with  some 
precision.  A  good  example  of  transfer  of  learning  is  the  educational 
setting  where  people  are  simply  being  educated  and  they  don't  know  precisely 
what  Job  they're  going  to  be  doing,  industrial  and  military  settings 
typically  Involve  transfer  of  training. 
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The  more  complex  the  system,  the  wore  difficult  It  becomes  to  fully 
•pacify  the  natura  of  the  performance  task.  The  basic  problem  Is  that  we 
have  to  try  to  estimate  tha  natura  of  tha  task  and  develop  a  training  task 
prior  to  tha  Implementation  of  tha  system.  Unfortunately,  there  eeems  to  be 
an  Infinite  number  of  transfer  tasks  that  can  be  created  for  any  given 
target  task. 

There  are  two  other  Interrelated  problems:  basic  research  findings 
tend  not  to  be  applied  effectively  to  real  world  situations  and  the  applied 
setting  le  not  necessarily  the  best  place  to  Investigate  basic  types  of 
psychological  processes  due  to  the  lack  of  experimental  control. 

I  will  present  some  of  my  own  observations  of  the  types  of  developments 
necessary  for  the  provision  of  these  types  of  training  estimates. 

First,  I  believe  that  the  analysis  of  tasks  should  be  based  on  the 
underlying  cognitive  and  psychomotor  processes  Involved  In  their  performance 
rather  than  using  the  task  procedures  as  our  classification  principle. 

For  example,  I  think  It  Is  very  Important  to  develop  a  consensus  on  the 
ways  In  which  we  can  analyze  tasks  on  ability  dimensions.  Can  we  come  up 
with  cognitive.  Information  processing,  and  psychomotor  factors  that  can 
then  be  used  to  analyze  any  given  transfer  task?  Some  of  these  factors 
might  Include  short  term  memory,  cuing  conditions,  and  decision  processes. 

It  seems  to  me  to  be  important  to  get  sway  from  defining  the  task  as  a 
set  of  procedures;  Instead  defining  It  as  a  particular  way  of  tapping  the 
Information  processing  or  psychomotor  abilities  of  the  subject.  Once  these 
analyses  have  been  done,  the  training  materials  could  be  developed  so  as  to 
simulate  the  task  specific  demands  of  those  dimensions. 

Second,  the  essence  of  being  able  to  estimate  or  analyze  tasks  Is  to 
use  these  Identified  factors  very  specifically.  For  Instance,  In  analyzing 
short  term  memory  demands,  are  there  portions  of  tasks  where  the  person  has 
to  keep  certain  Information  In  short  term  memory  In  order  to  be  able  to 
perform  the  task?  How  does  the  task  structure  affect  the  trainee's  use  of 
his/her  short  term  memory.  In  other  words.  This  Is  s  more  psychological  way 
of  doing  Job  analysis  than  Is  customarily  done. 

if  you  are  able  to  be  this  specific  about  task  analysis.  It  seems  it 
gives  us  much  more  power  In  the  training  environment.  This  specificity 
should  allow  us  to  be  more  analytical  In  Identifying  where  the  trainees  sre 
or  are  not  having  problems. 

it  le  difficult  to  investigate  the  same  types  of  Issues  In  an  applied 
setting  that  are  studied  In  a  basic  research  lab.  I  believe  that  applied 
research  Is  mors  useful  In  testing  the  practical  significance  of  factors 
■already  Identified  In  the  laboratory  setting. 

It  would  be  useful  to  consider  using  different  task  dimensions  In  the 
applied  environment,  for  ezample,  repeated  testing  or  extended  Intervals, 
which  are  often  not  done  In  the  laboratory. 
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Now  do  you  verify  the  validities  of  the  taxonomy  or  the  consensus  apart 
from  determining  that  the  concensus  la  a  valid  consensus?  How  do  we  know 
that  we  picked  the  right  taxonomy  or  the  right  dimension? 

The  Services  have  spent  a  great  deal  of  money  In  the  last  ten  or 
fifteen  years  trying  to  develop  tasks  particularly  based  upon  cognitive 
processes.  They've  been  largely  unsuccessful,  particularly  In  reliably 
attributing  certain  underlying  abilities  to  particular  tasks. 

For  example,  a  number  of  years  ago  the  American  Institutes  for  Research 
attempted  to  develop  some  scales  to  describe  underlying  motor  abilities,  and 
to  apply  those  scsles  to  a  particular  task  by  using  bench  marks  of  one  sort 
or  another.  The  results  generally  were  that  Judgments  were  not  very 
reliable. 

We  want  our  models  to  allow  us  to  specifically  Identify  what  Is  the 
effect  on  the  Individual  of  the  training  that  he  or  she  receives  In  terms  of 
target  task  performance.  For  example,  could  we  come  up  with  an  estimate  of 
whether  the  occurence  of  task  relevant  stimuli  would  exceed  the  capacity  for 
short  term  memory,  given  that  they  are  not  organized  by  differentiability, 
the  characteristics  of  recoding  from  verbal  to  visual  to  auditory 
modalities,  and  so  on? 

Another  example  would  be  In  the  area  of  automatization,  if  we  come  up 
with  a  series  of  tasks,  none  of  which  could  be  automatized  and  which  were 
being  required  to  be  simultaneously  performed  by  the  operator,  we  might 
predict  that  the  operator  would  have  difficulty  dealing  with  the  dual  task 
situation  unless  one  of  those  components  could  be  automatized. 


Some  might  suggest  using  the  expert  system  methodology  for  the 
development  of  these  models  of  cognitive  or  other  abilities.  Those 
individuals  who  are  knowledgeable  about  the  specific  systems  provide 
Information  that  would  be  used  for  preparing  the  content  of  the  training. 

Clearly,  expert  systems  can  be  useful,  however.  It  must  be  done  with 
great  caution.  There  is  potential  for  a  loss  of  control  In  the  development 
of  the  training  model.  How  much  control  can  we  give  subject  matter  experts 
In  defining  the  basic  cognitive  abilities,  for  example?  The  history  of  such 
attempts  suggests  that  In  this  complex  area,  task  experts  are  not  that 
reliable  In  their  Judgments. 

The  danger  In  trying  to  use  the  expert  to  lay  out  the  Instructional 
design  Is  that  their  own  understanding  of  how  they  are  able  to  perform  a 
task  and  which  abilities  they  employ  may  be  quite  limited.  As  l  see  It,  we 
have  no  alternative  to  the  effective  employment  of  the  findings  of  basic 
experimental  research  on  human  abilities. 
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One  thing  I  Ilka  about  this  meeting  la  that  there's  a  mix  of  people 
here  —  both  academic  and  "real-wor Id"  researchers.  I  think  I'm  fated  to 
represent  the  "unreal  world"  because  I'm  going  to  talk  about  those  things 
that  I  view  as  helping  to  "push  back  the  envelope”  of  what  we  know. 

I  have  a  particular  position  that  I  want  to  argue.  I  decided  that  l 
would  not  present  some  little  tidbit  of  my  research  but  Instead  try  and  show 
what  I  think  a  whole  domain  of  research  has  to  do  with  training  or  how 
training  could  be  done. 

I  want  to  discuss  the  role  of  cognitive  simulation  In  advanced  training 
technology.  The  thesis  Is  that  cognitive  simulation  modeling  Is  a  way  to 
obtain  the  specifications  for  what  must  be  learned.  This  can  then  provide 
the  Input  for  Intelligent  tutoring  and  other  advanced  training  systems  and 
also  low-tech  systems,  like  paper-based  training.  Cognitive  simulation 
modeling  Is  not  ready  to  be  put  Into  practice,  but  we  should  take  it 
seriously  as  a  potential  approach. 

I'm  going  to  talk  very  briefly  about  cognitive  simulation,  In  order  to 
define  It.  Then  I  want  to  describe  the  role  that  It  might  have  In  training. 

A  cognitive  simulation  la  a  computer  program  that  realizes  a 
theoretical  Idea  about  siental  structures  and  processes.  The  program 
contains  explicit  representations  of  mental  processes  and  knowledge 
structures.  State  changes  In  the  isodel  are  supposed  to  represent  state 
changes  In  the  mind  at  some  appropriate  level  of  analysis.  The  cognitive 
simulation  efforts  siost  commonly  Involve  symbol  manipulation  programming, 
especially  programming  written  In  the  LISP  language  using  artificial 
Intelligence  (A I)  programming  technlquea. 

Why  would  one  do  such  a  thing?  The  classical  reason  Is  that  this  Is  a 
way  to  discover  the  cognitive  processes,  because  this  approach  allows  you  to 
turn  vague  Ideas  Into  well-defined  Ideas.  This  approach  also  gives  you  the 
possibility  of  accounting  quantitatively  for  complex  empirical  data.  This 
Is  a  relatively  new  application  of  slawlatlon  modeling. 

As  psychologists  we  have  Ideas  that  are  often  vague  and  ill-defined. 
These  are  Ideas  that  guide  our  research  and  guide  our  thinking,  but  the 
scientific  activity  consists  of  trying  to  turn  those  vague  Ideas  into 
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•pec I f l c  Ideas.  A  simulation  Model  Is  •  tool  we  can  use  for  that.  It  gives 
a  lot  of  specificity  and  precision.  The  tradeoff  la  that  they  are  difficult 
to  construct  and  I  halted  In  appl lest  Ion. 


The  slaulat Ion  model,  when  properly  done,  is  In  some  sense  a 
realization  of  a  specific  set  of  psychological  theoretical  Ideas, 
no  point  In  building  one  of  these  Models  In  the  absence  of  good 
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psychological  Ideas;  In  the  Initial  use  of  simulation  Models  this  basis 
wasn't  Made  clear. 

This,  the  elMulatlon  Model.  Is  not  In  fact  the  psychological  theory. 
That's  a  different  set  of  Ideas.  But  It's  a  More  specific  version  of  the 
psychological  theories.  By  developing  the  slaiufatlon.  you  get  some  Ideas 
•bout  the  consistency,  the  completeness,  or  the  clarity  of  the  theory, 
because  you  have  to  be  specific  and  precise  with  a  computer.  You  are  forced 
to  turn  your  Ideas  Into  a  set  of  specific  Ideas. 

Such  simulation  models  currently  are  used  to  work  with  empirical  data 
at  levels  of  precision  end  detail  that  have  not  been  possible  In  the 
traditional  state  of  psychological  theory.  These  days,  with  our  laboratory 
computers,  we  can  collect  Much  more  detailed  data  than  was  formerly 
possible.  I  believe  these  simulation  Models  deal  with  data  at  that  same 
level  of  data  I f . 

Why  would  one  do  cognitive  simulation?  For  several  reasons:  You  want 
to  clarify  a  theoretical  Idea  by  constructing  an  explicit  realization  of  it. 
You  want  to  demonstrate  that  the  theoretical  Idea  you  have  Is  sufficient  to 
produce  the  behavior  of  interest.  The  history  of  psychology  Is  populated 
with  general  statements  about  things  that  rarely  were  shown  to  sctually  be 
capable  of  producing  the  behavior  that  people  were  talking  about. 

Sufficiency  Is  a  question  you  can  answer  In  a  very  straightforward  way 
with  a  simulation  model,  because  you  can  account  for  the  behavioral  data  in 
detail.  Once  you've  done  that  then  the  model  provides  you  with  this 
statement  that  summarizes  all  that  data.  So  Instead  of  looking  at  20 
graphs,  you  can  say.  "Here’s  the  Model  and  here's  what  It  does." 

What  does  a  simulation  have  to  do  with  training?  I  believe  It  Is 
possible  that  these  simulation  models  can  provide  theory-based  predictions 
and  evaluations  of  practical  situations. 

The  linkage  between  theoretical  Ideas  and  the  simulation  program  is 
bast  when  the  program  la  baaed  on  definite  assumptions  about  the 
architecture  of  cognition  —  the  basic  structure  of  what's  Inside  the  head  - 
-  as  opposed  to  just  having  code  that  produces  behavior.  This  seems  like  an 
obvious  point  but  It  hasn't  been  Made  explicit  before. 

There  Is  a  cognitive  architecture  that's  become  very  popular,  and  very 
important.  A  distinction  Is  made  between  declarative  knowledge,  knowledge 
of  facta,  and  procedural  knowledge.  Declarative  knowledge  Is  often 
represented  In  a  semantic  network,  which  Is  a  way  to  expreas  facts. 
Procedural  knowledge  Is  often  represented  by  a  set  of  production  rules  which 
examine  and  Manipulate  this  declarative  Information. 
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What  kind  of  Information  la  In  a  cognitive  simulation?  In  order  to 
specify  a  cognitive  simulation  model,  several  components  must  be  present. 

One  Is  a  detailed  task  analysis.  The  most  Important  part  of  this  analysis 
la  to  clarify  the  goals  this  simulation  model  or  the  human  must  accomplish. 

One  also  must  be  explicit  about  the  declarative  knowledge  that's 
required  to  do  the  task.  Are  there  certain  facts  about  the  domain  the  model 
has  to  have  In  order  to  be  able  to  do  the  task?  Likewise,  what  procedural 
knowledge  atust  the  model  have?  What  rules,  procedures,  and  heuristics  must 
It  have? 

As  an  example,  consider  electronics  troubleshooting.  If  we  had  a 
simulation  model  of  electronics  troubleshooting,  we  would  have  analyzed  the 
logical  structure  of  the  troubleshooting  task.  We  would  have  figured  out 
what  were  the  Important  facts  about  electronics  and  the  specific  system 
being  tested,  and  we  would  have  defined  what  were  the  various  rules  or 
heuristics  Involved:  for  example,  how  do  you  Infer  states  of  components 
from  observations? 

if  the  simulation  model  Is  successful,  then  we  have  some  measure  of 
confidence  that  we  have  accurately  and  completely  characterized  the  critical 
knowledge.  One  cannot  build  a  model  for  all  possible  tasks  in  a  particular 
domain.  But  once  we  have  a  model  to  do  one  thing,  at  least,  we  have  pretty 
complete  characterization  about  that  one  task. 

How  can  you  use  the  Information  In  a  model?  First,  the  task  analysis 
and  knowledge  epee  I f Icat Ions  characterize  the  task  Itself.  For  example,  you 
could  use  it  to  evaluate  the  quality  of  the  design  of  the  system  that  the 
trainee  must  Interact  with.  This  Is  In  essence  what  kleras  and  Poison 
(1085)  are  doing  In  the  field  of  human-computer  Interaction. 

You  can  use  It  to  compare  jobs  In  terms  of  the  underlying  tasks  because 
you  have  Identified  the  subtasks  that  are  Involved.  So  you're  In  a  position 
to  determine  which  Jobs  have  which  shared  tasks  If  you  build  enough  of  these 
models.  You  could  then  look  at  the  Job  and  find  out  how  much  information  a 
particular  Job  requires.  For  example,  you  might  discover  that  a  certain  Job 
Involves  only  a  couple  of  aubtasks  being  known  and  some  other  Job  might 
require  quite  a  few. 

The  specific  contents  of  the  declarative  and  procedural  knowledge 
provide  an  explicit  specification  of  the  minimum  required  knowledge  In  order 
to  do  the  task.  From  that  specification  you  can  tell  what  knowledge  is 
really  Important.  For  example,  the  classic  problem  In  trying  to  train 
people  In  electronics  maintenance  Is  an  Issue  about  what  electronics  theory 
they  really  need.  The  experienced  troubleshooter  might  say,  "Ohms  Law? 

Yes,  l  learned  that  once  and  never  use  It."  If  we  had  that  specification, 
we’d  be  In  a  position  to  make  much  more  Intelligent  guesses  about  what 
knowledge  people  actually  have  to  have  to  perform  a  task.  We  can  look  at 
the  courseware  to  see  If  It  contains  the  right  Information.  My  Intuition  is 
that  often  the  training  material  doesn't  have  the  Information  we  need  to 
make  a  computer  program  able  to  do  the  task. 


Finally,  If  you  have  these  explicit  representations,  you  have  now 
worked  out  a  lot  of  the  detail  necessary  to  do  things  like  build  intelligent 


tutoring  systems.  You  have  tho  character izat Ion  of  what  Must  be  learned  and 
you  can  export  It  to  other  kinds  of  systems. 

The  practicality  of  building  these  modele  presente  e  eet  of  "puzzles.1* 

Flret,  can  you  build  these  models  cost-effectively?  In  order  to  do 
that,  we  need  eome  kind  of  technology  of  cognitive  modeling;  Instead  of  an 
experienced  cognitive  psychoioglet  fiddling  with  one  for  a  year  or  two,  a 
graduate  student  In  cognitive  psychology  should  be  able  to  build  one  In  less 
than  a  year  because  they  would  know  what  they  had  to  do.  Such  technology 
requires  a  methodology  for  task  analysis.  If  we  could  specify  how  to  build 
one  of  these  modele  from  the  ground  up,  we'd  also  learn  a  lot  about  how  to 
do  this  cognitive  task  analysis  we've  been  talking  about.  The  modeling 
approach  provides  you  with  a  way  of  verifying  the  task  analysis  on  a  small 
scale. 

A  second  question  Is  whether  the  really  complex  Issues  can  be  finessed, 
in  some  sense  we  shouldn't  have  to  model  everything  about  a  task.  We 
shouldn't  have  to  construct  an  entire  model  of  perception,  for  example,  In 
order  to  construct  a  model  of  how  people  do  electronic  troubleshooting, 
because  regardless  of  how  the  perceptual  system  works.  It  does  deliver 
certain  outputs  to  the  rest  of  the  system.  We  want  to  know  what  Is  in  the 
rest  of  the  system. 

So  a  useful  model  would  be  one  that  had  a  lot  of  these  perceptual 
Issues  that  you  could  finesse,  by  saying  the  Input  to  the  system  Is  the 
simpler  output  of  the  perceptual  system. 

But  can  we  reliably  identify  and  focus  on  Just  that  critical 
Information?  Do  we  know  how  to  finesse?  Do  we  know  how  to  choose  what  to 

finesse? 

One  of  the  more  valuable  things  about  this  approach  is  that  the 
Information  In  the  cognitive  model  Is  also  the  Information  we  want  In 
Intelligent  tutoring  or  an  advanced  testing  system.  How  do  we  streamline 
the  export  of  Information  from  one  of  those  efforts  to  the  next?  One  or  the 
things  you  have  to  do  Is  work  with  compatible  knowledge  representations.  It 
would  be  easier  if  there  was  a  general  consensus  on  the  cognitive 
architecture  we  want  to  assume.  Some  examples  of  how  this  approach  can  be 
applied  can  be  found  In  a  recent  technical  report  (Kleras,  1087,  in  press). 
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In  my  presentation  I  will  describe  a  method  for  characterizing  the 
dimensions  of  transfer  space  and  then  talk  about  how  that  characterization 
can  be  translated  into  a  specification  of  transfer  objectives  that  can  serve 
as  the  basis  for  training  procedures.  My  Ideas  are  speculative  and  I  would 
appreciate  your  thoughts  on  them. 

There  has  been  aiuch  discussion  today  about  the  various  situations  In 
which  transfer  problems  appear.  My  talk  will  be  devoted  to  developing  a  way 
of  characterizing  those  situations  and  to  Identifying  the  kinds  of  problems 
that  one  might  conceptualize  as  transfer  problems. 

A  Two  Dimensional  Representation  of  Transfer  Space. 

One  way  to  characterize  a  transfer  situation  is  to  specify  the 
relationship  between  the  physical  similarity  of  the  instructional  task  and 
the  transfer  task.  This  Is  the  old  "identical  elements"  notion:  the  idea 
that  transfer  Is  going  to  be  determined  by  the  degree  of  stimulus  similarity 
between  the  training  task  and  the  transfer  task.  In  the  discussions  thus 
far  several  speakers  have  spoken  of  the  "fidelity"  of  training  to  Job 
performance;  this  Is  one  kind  of  Idea  I  am  trying  to  capture  In  my 
characterization  of  the  Identical  elements  dimension  of  transfer  space. 

Identical  elements  theory  only  takes  you  so  far  In  characterizing 
transfer  tasks  and  I  would  like  to  suggest  that  there  is  a  second  dimension 
of  transfer  tasks  that  Is  as  Important  as  the  physical  similarity  dimension. 
That  distension  Is  the  complexity  of  the  knowledge  that  Is  being  used  to 
perform  the  task. 

In  recent  years  two  advances  have  given  us  better  ways  of 
characterizing  the  knowledge  distension  of  transfer  tasks.  One  advance  is 
the  research  on  differences  between  expert  and  novice  performance  that  has 
emphasized  the  fact  that  knowledge  differences  between  the  expert  and  novice 
are  as  much  qualitative  as  they  are  quantitative.  The  second  advance  is  the 
utilization  of  production  systems  as  the  bases  for  theories  of  cognition. 
Production  systems  provide  concrete  models  of  processing  events  that  In 
earlier  theories  were  labeled  boxes  In  a  diagram. 

My  notion  of  character Izlng  transfer  space  using  the  dimensions  of 
stimulus  similarity  and  knowledge  complexity  Is  represented  In  Figure  1.  A 
condition  of  training  or  Instruction  Is  represented  at  the  far  left 


Description  of  Transfer  Space 


Intercept  of  the  graph  In  Figure  1.  The  horizontal  ax  la  of  the  graph 
representa  the  degree  to  which  the  tranafer  task  la  dissimilar  (in  a 
stlmulua  eenae)  from  the  condltlona  of  original  Inatruetlon.  For  example, 
close  to  the  point  of  original  Inatruetlon  on  the  horizontal  axle  are  tasks 
that  closely  resemble  those  that  were  considered  during  Instruction  or 
training.  At  the  extreme  right  on  the  horizontal  axis  are  tasks  that  can  be 
completed  utilizing  the  knowledge  acquired  In  training  but  these  tasks  are 
quite  different  In  a  physical  or  stimulus  sense  from  the  conditions  of 
original  Instruction. 

The  vertical  axis  of  the  graph  represents  the  complexity  of  the 
knowledge  that  can  be  utilized  to  perform  a  particular  task.  The 
characterization  Is  from  very  simple  or  low  level  knowledge  to  very  complex 
high  level  knowledge.  I  want  to  emphasize  that  the  notion  of  knowledge  that 
l  am  trying  to  capture  le  not  degree  of  knowledge,  but  rather  kind  of 
knowledge.  What  I  have  In  mind  Is  the  sorts  of  differences  that  have  been 
discovered  between  expert  and  novice  physicists.  The  expert  physicist 
appears  to  have  his  or  her  knowledge  represented  In  hierarchical  structures 
with  the  apex  of  the  structure  consisting  of  very  high  level  principles  that 
encompass  an  enormous  amount  of  subsidiary  Information.  That  kind  of  high 
level  knowledge  Is  what  I  am  trying  to  represent  at  the  upper  extreme  of  the 
vertical  axis  In  Figure  1. 

Within  the  graph  l  have  provided  eome  labels  that  night  give  you  some 
Idea  of  the  kind  of  performance  1  am  trying  to  capture  with  my  two 
dimensions  of  transfer  tasks.  Within  the  lower  left  quadrant  of  the  graph 
we  have  performance  of  tasks  that  require  relatively  low  levels  of  knowledge 
and  that  are  physically  similar  to  those  encountered  In  original 
Instruction.  In  the  lower  right  quadrant  are  tasks  that  require  relatively 
low  levels  of  knowledge  but  the  transfer  situation  Is  quite  different  from 
the  conditions  of  original  Instruction.  In  the  upper  left  quadrant  we  find 
expert  performance  —  the  performance  of  tasks  requiring  the  use  of  complex 
knowledge  but  still  within  the  context  of  those  conditions  that  existed 
during  original  learning  or  training.  Finally,  In  the  upper  right  quadrant 
we  have  the  performance  of  tasks  that  require  very  high  level  knowledge  in 
situations  that  are  very  dissimilar  from  the  conditions  that  existed  during 
original  learning.  This  is  the  sort  of  performance  that  we  might  expect 
from  people  who  make  scientific  breakthroughs. 


If  the  way  of  characterizing  transfer  tasks  that  l  have  described  Is  to 
have  any  practical  utility  It  must  be  possible  to  objectively  describe 
levels  of  both  dimensions.  1  would  like  to  spend  the  next  few  minutes 
talking  about  how  this  might  be  done.  1  will  begin  with  an  example  of  how 
we  might  characterize  knowledge  about  learning  theory.  It  might  go 
something  like  the  representation  I  have  depicted  In  Figure  2.  As  we  go 
from  the  upper  sest  node  that  I  have  called  "learning  theory”  the  hierarchy 
spreads  Into  two  nodes  that  l  label  "cognitive  theory"  and  "behavioral 
theory."  Under  cognitive  theory  I  branch  Into  Inforstatlon  processing 
theory,  schema  theory,  and  Gestalt  theory.  In  behavioral  theory  l  have 
social  learning  theory,  operant  theory,  and  associative  learning  theory.  At 
the  next  level  down  I  have  specific  theorists  such  as  Atkinson,  Shank,  Hull, 
and  Skinner.  At  a  lower  level  yet  I  have  specific  phenomena  and  specific 
experimental  rssulte. 
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Figure  2 

Knowledge  Hierarchy  for  Learning  Theory 
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I  would  not  argue  that  the  hierarchy  I  have  depleted  in  Figure  2  Is  an 
agreed  upon  way  of  claesifylng  thle  particular  knowledge  domain,  but  I  would 
suggest  that  Is  possible  for  a  group  of  experts  to  come  to  some  agreement 
with  respect  to  characterizing  the  knowledge  hierarchy  present  In  a  narrowly 
constrained  knowledge  domain  such  as  learning  theory. 

One  aspect  of  the  knowledge  hierarchy  as  It  relates  to  expert/novice 
performance  le  that  when  an  expert  Is  presented  with  some  phenomena  related 
to  learning  theory  the  expert  activates  very  high  level  knowledge  units 
automatically  as  a  function  of  encountering  phenomena.  Some  cognitive 
theorists  have  referred  to  this  as  the  procedural  Hat  Ion"  of  knowledge.  I 
want  to  emphasize  that  the  structure  I  have  presented  le  not  a  model  of  how 
the  content  within  a  domain  should  be  taught.  I'm  reasonably  certain  that 
teaching  a  novice  all  the  labels  and  connections  In  a  knowledge  hierarchy 
would  have  little  latpact  on  the  novice's  ability  to  classify  phenomena  that 
were  very  different  than  those  encountered  previously. 

The  particular  knowledge  hierarchy  that  I  represent  In  Figure  2  happens 
to  be  one  associated  with  a  cognitive  domain.  But  It  le  also  possible  to  do 
a  similar  kind  of  analysis  for  certain  kinds  of  motor  skills.  For  example. 
In  tennis  one  can  divide  skills  Into  shot  skills  and  strategy  skills.  Shot 
skills  are  mastery  of  forehand,  backhand,  volley,  etc.  Strategy  skills  come 
Into  play  In  decisions  about  when  to  utilize  particular  shots.  One  of  the 
distinctions  between  a  truly  expert  tennis  player  and  a  good  club  player  is 
that  the  club  player  may  have  good  shots  but  he  doesn't  have  the  strategy 
components  of  the  game.  The  expert  player  not  only  hits  great  shots  but  he 
hits  the  right  shot  at  the  right  time;  that's  what  percentage  tennis  Is  all 
about . 

The  thing  that  Is  common  In  my  character Izat Ion  of  the  knowledge 
hierarchies  of  the  expert  learning  theorist  and  the  expert  tennis  player  is 
that  a  particular  event  In  either  domain  automatically  triggers  off  very 
high  levels  of  knowledge  In  the  domain.  In  contrast,  the  same  event 
triggers  off  lower  levels  of  knowledge  In  the  novice. 


If  one  accepts  the  assumption  that  the  same  phenomena  triggers  off 
knowledge  at  different  levels  in  the  expert  and  the  novice,  this  has 
Implications  for  the  characterization  of  transfer  apace  that  I  presented  in 
Figure  1.  Consider  people  who  master  information  at  various  levels  on  the 
knowledge  level  axle  of  Figure  1.  Now  Imagine  a  range  of  transfer  tasks  the 
knowledge  will  transfer  to.  My  hypothesis  Is  that  someone  mastering 
knowledge  at  a  relatively  low  level  of  knowledge  will  only  be  able  to 
transfer  that  knowledge  to  a  small  range  of  transfer  tasks  that  have  a 
stimulus  context  that  le  similar  to  the  conditions  of  original  Instruction. 
In  contrast,  the  person  who  has  oastsrsd  knowledge  at  a  high  level  could  be 
expected  to  transfer  that  knowledge  to  a  much  broader  range  of  tasks.  I 
have  depicted  this  prediction  In  Figure  3.  This  suggestion  leads  to  a 
further  hypothesis.  If  you  want  to  train  for  transfer  you  can  do  It  In  one 
of  two  ways:  Expand  the  range  of  stimulus  characteristics  the  trainee  Is 
exposed  to  during  training,  or  extend  the  complexity  of  the  knowledge  the 
trainee  le  acquiring. 


I  envision  the  knowledge  domains  I  am  talking  about  to  be  narrowly 
constricted  with  a  vertical  rather  than  a  horizontal  architecture.  That  is. 
acquired  knowledge  within  a  domain  can  transfer  downward  to  related 
situations,  but  Information  will  not  transfer  across  domains.  For  example, 
chess  skills  are  unlikely  to  transfer  to  card  or  board  games. 

The  two  dimensional  characterization  of  transfer  space  that  l  have 
described  could  be  used  in  planning  training  that  will  transfer  to  other 
situations.  The  Idea  Is  that  the  instructional  developers  first  must  decide 
what  trainees  should  be  able  to  do  after  completing  training.  This 
description  of  what  they  should  be  able  to  do  would  Include  a  description  of 
the  range  of  conditions  the  training  should  transfer  to  (you  could  think  of 
these  as  transfer  objectives).  Let's  examine  several  Instances  that  we  can 
represent  on  Figure  1.  Suppose  that  an  Individual  Is  being  trained  to 
repair  field  radios.  Further,  let's  suppose  that  the  radios  the  Individual 
Is  being  trained  to  repair  all  look  pretty  much  the  same  and  they  have 
similar  design  and  function.  This  situation  could  be  represented  in  Figure 
1  by  Imagining  a  transfer  space  very  close  to  the  point  representing 
original  Instruction  on  the  graph.  An  Individual  being  trained  to  perform 
the  function  of  repairing  the  field  radios  could  probably  perform  his  or  her 
Job  successfully  by  being  given  a  minimal  amount  of  "textbook"  training  on 
electronic  theory  and  troubleshooting  and  then  practical  training  on 
actually  repairing  "prototype"  field  radios. 

Now  Imagine  a  situation  where  an  Individual  Is  being  trained  to  repair 
a  wide  variety  of  electronic  equipment.  In  other  words,  the  training  is 
expected  to  transfer  to  situations  on  the  far  right  of  Figure  1.  Now 
training  can  either  consist  of  relatively  low  level  abstract  learning  and 
considerable  practical  experience  on  a  wide  variety  of  equipment  (l.e., 
expanding  the  physical  similarity  dimension  of  training),  or  It  can  consist 
of  focusing  on  higher  levels  of  knowledge  that  will  also  result  In  the 
expansion  of  the  range  of  problems  the  trainee  can  successfully  handle. 

It  seems  to  me  that  the  trick  In  carrying  out  the  kind  of  analysis  I  am 
describing  Is  determining  exactly  what  you  would  like  a  trainee  to  do. 

There  are  probably  many  situations  In  military  training  where  trainees  are 
only  going  to  be  faced  with  Job  situations  that  can  be  virtually  duplicated 
In  the  training  environment.  In  this  kind  of  case  low  level  knowledge 
training  coupled  with  practical  experience  would  probably  accomplish  the 
training  goal.  In  other  situations,  however.  It  may  be  very  difficult  to 
specify  the  range  of  tasks  the  trainee  will  be  required  to  accomplish  in  the 
Job  situation.  In  this  case  the  trainee  should  be  prepared  for  the  maximum 
transfer  potential  which  means  the  acquisition  of  complex  knowledge 
accompanied  by  experience  with  a  range  of  tasks.  A  further  complication 
associated  with  specifying  the  range  of  situations  that  one  would  expect 
training  to  transfer  to  Is  that  Job  requirements  change  over  time.  The 
Individual  who  has  been  trained  to  perform  one  kind  of  task  on  a  limited 
range  of  equipment  may,  over  time,  find  that  the  requirements  of  the  Job 
•have  changed  and  the  Initially  received  training  Is  no  longer  adequate  for 
Job  performance. 


In  my  talk  today  I  have  presented  a  means  of  describing  transfer  space 
In  terms  of  two  distensions:  a  distension  describing  the  physical  similarity 
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I  want  to  contrast  practical  notions  of  transfer  with 
laboratory/research  notions.  This  is  a  contrast  between  schooling  and  what 
i'll  call  •aientor ing". 

‘Schooling*  Is  the  experience  that  aost  psychologists  have  that  gives 
us  a  particular  aiental  model  that  we  use  to  deal  with  the  world. 

Schooling  Is  a  situation  with  a  general  goal  —  to  build  an  educated 
person  through  elementary  school,  secondary  school,  and  other  schooling. 
Schooling  Is  not  tied  Into  practical  objectives  for  the  most  part,  and  in 
fact.  If  you  listen  to  educators  argue  about  curriculum  and  curriculum 
planning,  you  find  that  the  arguments  focus  on  how  to  specify  the  generic 
goals  of  education.  The  goals  are  quite  abstract.  Yet,  because  we  worry 
about  transfer,  we  look  at  these  abstractions  as  If  we  can  do  something 
about  them.  For  example,  "can  we  train  generic  problem-solving  skills”? 

Can  we  train  the  components  of  reading  If  there  are  these  components,  like 
decoding  skills  and  certain  vocabulary  access?  If  we  train  those  skills, 
will  they  transfer  and  make  everybody  better? 

Unfortunately,  generic  training  programs  don't  seem  to  work  well,  and 
because  of  the  level  of  abstraction,  1  would  argue  that  there  Is 
considerable  evidence  that  shows  a  lot  of  this  Is  difficult  for  students. 

It's  difficult  for  students  to  learn  abstractions  (e.g.,  generic  rules 
for  solving  problems)  that  are  separate  from  concrete  examples.  Research  in 
cognitive  psychology  has  shown  that  you  need  concrete  examples  to  facilitate 
learning. 

Many  of  the  abstract  things  that  people  are  supposed  to  learn.  In 
geometry  for  example,  are  not  made  explicit  In  the  Instructional  situation. 
They  are  always  to  be  discovered,  to  be  uncovered,  and,  of  course,  most 
students  never  do  uncover  them. 

There's  a  need  for  explicitness.  The  science  education  literature 
provides  evidence  that  there's  weak  or  little  transfer  between  the  academic 
.  abstractions  learned  In  class  and  practical  situations  where  you  have  to 
transfer  (use)  that  knowledge  to  solve  problems. 

DISsssa  demonstrated  fairly  conclusively  with  physics  that  what  we  get 
out  of  the  educational.  Instructional,  or  training  process  Is  a  separate 
body  of  academic  knowledge  that  applies  In  class  but  not  In  situations  where 
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that  knowledge  should  transfer.  The  notion  le  that  Information  doesn't 
transfer  because  the  connections  between  the  abstract  formal  Isms  and 
concrete  events  are  not  made  properly. 

Most  psychologists  and  educators  have  the  viewpoint  that  this  schooling 
model  Is  the  kind  of  model  we  expect  In  a  learning,  training,  or  the 
Instructional  situation.  This  Influences  two  things.*  It  Influences  what 
research  they  do  and  the  way  In  which  they  begin  to  think  about  transfer  and 
transfer  of  training;  and  It  Influences  the  Instruction  they  design  and,  l 
think,  fosters  the  lack  of  attention  to  the  strategic  processes  that  are 
part  of  the  Instruct  Iona  I  process. 

If  we  look  at  the  "mentoring"  situation  — ■  or  perhaps  a  better  term 
would  be  "experiential"  situation  —  we're  really  talking  about  a  vocational 
or  apprenticeship  aiodel.  It  Is  involved  with  domain  specific  performance. 
The  result  of  the  Instruction  process  Is  to  build  a  domain  theory  Into  the 
Individual,  to  guide  his  or  her  perforawnce.  The  only  tlam  you  need  to 
teach  the  individual  about  formal  theory  is  when  that  theory  la  directly 
relevant  to  what  he  or  she  Is  going  to  undertake.  The  practice  given  Is 
concrete  and  very  specific.  It  Is  usually  also  Implicit  that  It's  a 
one-on-one  Instructional  situation.  In  that  situation  strong  transfer  is  a 
primary  objective  but  It  Is  of  the  kind  that's  been  called  "near"  on  a 
"near-to-far •  transfer  dimension. 

We  need  to  do  a  more  adequate  analysis  of  the  strategic,  metaphorical 
and  analogical  knowledge  needed  for  far  transfer. 

I  want  to  contrast  teaching  abstracted  components  of  skill  with 
simulation,  because  from  the  practical  viewpoint.  If  I  need  to  train  people 
to  do  particular  jobs,  the  primary  emphasis  will  be  to  mimic  situation 
characteristics,  or  use  simulations,  of  the  actual  Job. 

If  I'm  going  to  do  that  In  school.  I'm  going  to  push  toward  the  use  of 
simulation  and  role  playing.  I  don't  care  whether  It's  paper  and  pencil 
simulation  or  computer-based  simulation,  but  the  emphasis  le  on  how  I  can 
represent  critical  functional  aspects  of  the  real  task. 

Up  front  Is  the  Issue  of  realistic  simulation,  not  analysis  into 
psychological  components.  When  we  talk  about  generic  notions  of  transfer, 
we  get  involved  across  domains  and  require  abstract  conceptions  about  the 
skill  that  transfers. 

There  le  confusion  between  the  specific  aspects  of  tasks  and  the 
strategic  knowledge  about  how  you  act  In  this  particular  situation,  l.e., 
general  approaches  to  the  world.  For  instance,  your  chess  ski  I  I/knowledge 
might  apply  to  "Co"  or  checkers. 

Notice  that  In  crossing  domains  the  focus  le  on  generic  processes. 

When  you  start  with  a  Job-specific  viewpoint,  the  focus  has  to  be  on 
representing  that  domain  to  a  student,  not  teaching  abstract  rules. 

One  of  the  lessons  the  people  from  the  "schooling  camp"  might  consider 
le  how  to  describe,  analyze,  represent  and  model  the  "Jobs"  that  students 
might  be  expected  to  do.  Deriving  instructional  situations  from  these 
descriptions  Is  simpler. 


The  Military  ha*  adopted  a  aystaai  planning  Model  or  procedure  for 
attempting  to  relate  training  to  the  job.  You  derive  the  training 
practlcea,  the  deeerlptlona  of  what  you  mean  by  competence,  and  how  you  will 
go  about  teatlng  all  from  ana l ya l a  and  repreaentat Ion  of  Job  teaks.  That's 
the  theory,  and  the  way  the  proceae  la  eet  up.  It  Is  a  sound  set  of  Ideas 
and  can  be  effective. 

This  procedure  le  not  Implemented  well.  People  adopt  an  academic  model 
of  schooling  rather  than  a  mentoring  model.  Let  me  Illustrate  the  point. 

«e  looked  at  a  course  that  was  part  of  the  propulsion-engineer Ing 
curriculum.  The  course  Is  supposed  to  teach  the  student  how  to  do  a 
chemical  analysis  of  water  that  comes  out  of  the  condensers  In  the  boiler 
system. 

You  would  espect  the  curriculum  to  Include  come  very  simple  basic 
chemistry  and  procedures.  Instead,  they  have  a  full  two-week  course  In 
chemistry  —  probably  far  more  than  Is  necessary.  Again,  the  schooling 
model  Imposes  Itself  on  the  training  developers  and  they  then  don't  follow 
the  mentor /exper lent  la  I  model  very  well. 

I  have  some  recommendations  for  research  and  development.  First,  there 
Is  a  history  of  20  or  30  years  of  doing  task  analysis  systematically  In  the 
military  and  the  Industrial  world.  That  process  la  only  partially  adequate, 
because  the  strategic  knowledge  and  the  organization  aspect  of  the  task  are 
not  in  that  aodel.  There  should  be  research  to  Improve  that  process. 

In  all  Instructional  situations  we  allude  to  the  entering  student's 
knowledge.  We  use  selection  tests  and  perhaps  some  specific  knowledge 
tests,  but  none  of  these  are  necessarily  tystemat leal ly  related  to  the 
course.  If  It  le  true  that  a  student  l»arn$  well  on  the  basis  of  his  old 
knowledge,  which  Is  what  many  cognitive  scientists  would  have  us  believe, 
then  it  le  Important  to  know  what  that  "old  knowledge"  Is  to  adapt 
Instruction  appropr lately . 

There  Is  a  problem  of  Instructional  fidelity.  Mow  do  we  take  apart  the 
critical  aspects  of  the  task  to  Identify  what  pieces  (components)  of  the 
task  should  be  the  ones  that  get  practiced  during  the  schooling? 

Performance  measurement  takes  too  much  time  and  energy  to  be  practical. 
Therefore,  if  I  ever  want  to  measure  competence  or  even  expertise,  how  do  i 
measure  its  level?  In  chess  and  alollar  domains,  there  are  competitive 
systems  where  people  have  to  deal  with  one  another  to  get  master  points. 
That's  how  they  are  rated  one  against  the  other.  Mow  do  we  do  that  In  other 
situations  without  taking  lots  of  time? 

I  think  It  la  worth  emphasizing  that  the  people  who  are  going  to  use 
this  knowledge  are  not  the  people  In  this  room.  Instructors  and  teachers  in 
the  real  world  are  the  ones  who  have  to  apply  what  results  from  better 
system l zat Ion.  How  do  we  reach  them?  We  need  automated  Implementation  aids 
to  help  with  the  analysis,  the  design  process,  and  the  design  of  evaluation 
systems.  We  need  automated  systems  that  Include  cognitive  and  psychological 
knowledge. 


Finally.  I  think  that  Implementation  of  systematic  Instructional 
development  processes  must  be  Improved.  A  feedback  and  monitoring  system 
for  the  whole  process  Is  essential.  The  model  I  have  for  that  feedback  and 
monitoring  system  was  described  by  Bill  Cooley  In  his  Presidential  address 
at  an  annual  meeting  of  the  American  Educational  Research  Association  a 
couple  of  years  ago,  where  he  talked  about  a  model  he  built  for  the 
Pittsburgh  school  system.  Cooley's  model  takes  measures  out  of  the  school 
system.  Identifies  a  problem,  and  then  Institutes  changes  with  teams  of 
Individuals  who  would  solve  the  problems  via  appropriate  actions.  I  think 
the  same  sort  of  thing  can  be  established  for  other  situations. 

The  Navy  has  a  feedback  system  In  which  Job  supervisors  of  recent 
graduates  complets  a  questionnaire  that  Is  related  to  Job  objectives.  This 
system  doesn't  work  very  well  and  Is  Incomplete,  but  It  does  Identify 
certsln  real  problems. 

The  questionnaire  asks  If  the  student,  when  he  first  arrived  from 
school,  could  do  a  certain  task,  it  Is  a  partial  system  —  Just  a  survey. 

It  does  not  have  a  feedback  loop  In  It  to  Identify  a  problem  and  tie  It 
closely  to  the  training  process.  Thsse  supervisors  are  reporting  on 
students  several  months  sftsr  they've  arrived,  but  they're  supposed  to 
reference  them  In  time  to  when  they  sr rived,  because  you  don't  want  to 
confound  it  with  whatever  they've  learned  In  the  Job  situation. 

In  the  military  In  a  peace  time  world,  relatively  few  people  with  a 
poor  academic  background  are  recruited.  Under  peace  time  pressures  to  keep 
expenditures  down,  selection  of  higher-level  people  Is  cost-effect ive. 

If  we  get  Into  a  wartime  mobilization,  all  of  a  sudden  we  accept  lots 
of  people  who  we  would  screen  out  right  now;  training  those  people  Is  more 
difficult.  How  csn  we  train  these  people  more  quickly? 

Many  issues  of  transfer  have  to  focus  on  that  relatively  low  end  —  t 
don't  like  to  use  the  word  "Intelligence,”  but  I  don't  know  what  else  to  use 
for  that  dimension.  Work  was  done  on  this  problem  during  the  second  World 
War,  the  Korean  War,  and  more  recently  In  "Project  100,000,”  when  100,000 
people  were  taken  and  the  military  had  to  worry  about  whether  they  could 
utilize  them  In  technical  areas  for  which  they  were  at  least  nominally 
unqualified.  From  recent  analyses  they  performed  nearly  as  well  as  higher 
ability  people. 

Military  training  Is  always  going  to  be  much  less  generic  than  ordinary 
education.  For  example,  the  Navy  has  Military  Occupational  Specialties 
(M0S),  which  are  defined  as  a  group  of  Job  sub-tasks.  Training  Includes  a 
general  basic  course,  then  more  specific  course(s)  which  tend  to  be  very 
equipment  oriented.  The  fundamentals  course  Is  supposed  to  provide  the 
basic  knowledge  and  skills  required  for  that  Job.  Even  that  Is  an  academic 
model,  but  It  Is  tied  mueh  more  closely  to  the  Job  In  the  design  of  the 
curr Iculum. 

For  example,  the  fire  control  technician,  who  Is  Involved  in  shooting 
missiles,  and  fixing  those  systems  that  shoot  missiles.  Is  going  to  learn 
about  fire  control  and  about  particular  systems  before  working  on  these 
systems. 


It  Is  the  fundamental  (first)  course  that  becomes  Nice  other  academic 
courses.  The  fact  that  these  generic  courses  are  more  "schooling"  Is  of 
concern.  They  should  be  briefer  and  more  Job-1  Ike  eo  that  the  trainee  can 
fix  the  gear  and  be  useful  right  sway.  "Experiential"  courses  would  be 
better.  If  we  could  develop  them.  In  this  model,  the  student  la  given  job- 
llke  experience  and  Is  brought  to  the  Job  as  an  apprentice  to  help  him  pick 
up  the  fine  arts.  He  has  a  period  of  time  on  the  Job  with  much  less  well 
controlled  supervision  that  would  be  the  case  In  his  schooling.  That's 
where  you  would  begin  to  see  the  transfer  of  training  —  In  his  ability  to 
learn  on  the  Job.  Current  training  Is  too  acaderic  and  does  not  prepare  him 
for  learning  on-the-job  well. 

Because  of  marginal  training,  he  doesn't  get  a  chance  to  refresh 
himself  on  some  of  the  things  that  were  explicitly  Included  In  his  training 
courses.  Often,  the  guy  who  most  needs  training  will  spend  the  day  sw eeplng 
floors,  because  they  can't  afford  to  risk  practice  on  the  equipment.  Many 
people  come  out  of  basic  training  courses  without  having  acquired  the 
"basic"  skills.  For  example,  there's  not  enough  practice  soldering,  not 
enough  practice  using  test  equipment,  and  no  systematic  training  In  how  to 
find  Intermittent  connections  In  cables. 

Experiential  models  are  difficult  to  Implement,  due  at  least  In  part  to 
resource  constraints.  The  model,  although  an  old  one.  Is  still  good,  in 
fact,  Prosser  and  Quigley  published  a  book  In  1B25  suggesting  such 
principles  for  teaching.  It  was  more  cognitive  than  behavioristic  at  that 
time.  The  environmental  context  and  the  situational  context  variables  were 
very  Important  for  the  Instructional  process.  You  had  to  represent  to 
students  the  real  situation  Insofar  as  possible  and  give  them  experience 
functionally  equivalent  to  the  job. 

If  we  have  certain  tools  for  developing  Instruction,  we'd  have  a  better 
chance  of  Implementing  these  models.  For  Instance,  we  are  working  on  an 
under-funded  project  on  building  an  Intelligent  authoring  system.  The  Idea 
Is  to  provide  an  environment  In  which  someone  who  Is  beginning  development 
of  Instruction  —  even  In  the  analytic  process  —  would  be  aided  and  advised 
by  this  system.  You  can  think  of  It  as  an  advisor  or  coach,  but  It  would 
also  help  carry  out  the  mundane  tasks,  such  as  the  generation  of  textual 
materials  and  graphics. 

it  would  also  ultimately  have  a  rapid  prototyping  system  so  that  the 
Individual  could  see  the  representation  that  he  wanted  to  show  to  his 
student  and  try  It  out,  and  If  he  didn't  like  It,  he  could  throw  It  away  and 
develop  a  new  one  quickly.  You  can't  do  that  kind  of  thing  very  quickly 
today,  and  people  like  Jim  Hoi  Ian  at  NPRDC  and  Doug  Towns  at  USC  and  others 
are  working  to  build  devices  that  are  pieces  of  this  ultimate  system. 

One  Issue  In  developing  prototype  simulations  for  training  Is  what  I 
-call  "Instructional  fidelity."  People  who  develop  simulators  emphasize 
fidelity  to  the  task  and  equipment  actually  used.  Such  devices  are  costly 
and  may  not  be  the  best  means  of  training  because,  in  representing  the  task 
In  all  Its  complexity.  It  Is  Inefficient  for  student  learning. 

Instructional  fidelity  Is  concerned  with  how  to  Implement,  for  example, 
part-taek  training,  backward  chaining,  time  compressed  practice.  Such 
procedures  are  needed  to  make  Instruction  more  effective  and  efficient. 


Alan  Lesgold,  Ph.D 


Learning  Research  and  Development  Center 
University  of  Pittsburgh 


I  want  to  relate  eome  experiences  that  my  colleagues  and  I  have  had  in 
looking  at  several  specific  technical  areas,  and  In  trying  to  determine  how 
training  might  be  Improved  In  these  situations. 

Second,  I  want  to  say  a  few  things  about  the  kinds  of  knowledge  that  I 
think  are  needed  to  drive  Intelligent  training  systems  or  expert  training 
systems.  Finally,  I  want  to  comment  on  certain  misconceptions  and  the  role 
they  might  play  In  thinking  about  transfer. 

We'll  start  by  looking  at  the  training  of  avionics  technicians  In  the 
Air  Force. 

There's  a  kind  of  "high-tech*1  trap  that  exists  In  training  these  days. 
People  come  along  with  products  to  sell  to  the  DoD,  or  people  within  the  DoD 
who  want  particular  products  shape  their  definition  In  ways  that  make  the 
products  look  as  good  as  possible. 

Inevitably,  somewhere  along  the  line  we  come  to  believe  that  the 
devices  we  bought  are  really  simple  —  so  simple  that  anyone  could  fix  or 
use  them.  That  effect  is  one  of  the  reasons  why  certain  kinds  of  specific 
procedural  training  find  their  way  Into  the  training  process. 

For  many  systems  there  are  books  that  purport  to  describe  exactly  what 
to  do  In  every  circumstance,  so  that  the  real  task  Is  to  train  people  to 
follow  the  Instructions,  to  do  so  quickly,  and  without  being  too  original  in 
the  Implementation  of  those  Instructions. 

The  problem  Is  that  It  doesn't  work  that  way.  The  systems  are  never  as 
foolproof  as  we  might  have  hoped,  and  that  In  turn  produces  a  problem.  We 
train  a  person  to  follow  a  very  specific  Instruction,  and  suddenly  we 
realize  he  has  to  do  some  additional  tasks  just  to  survive.  But,  he  was 
taught  only  Instruction-following,  rather  than  some  store  flexible  knowledge 
that  might  have  helped  him  deal  with  complex  cases. 

The  example  1  want  to  cite  Involved  people  who  used  a  relatively 
elaborate  test  station  to  fix  the  little  boxes  of  electronic  stuff  that  they 
stick  In  planes.  The  stuff  In  those  little  boxes  Is  very  reliable,  but  the 
test  stations  have  been  around  a  long  time  and  are  not  as  reliable. 

It  Is  Important  to  note  that  planes  that  sre  used  In  the  Air  Force  have 
been  designed  relatively  recently.  They  undergo  a  certain  amount  of 
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modification  over  time.  Often,  when  plane  for  new  planet  are  made,  a 
reliable  teat  etatlon  le  Included;  but  ee  budgetary  preaeures  arise,  the  new 
teat  atatlon  often  dleappeare. 

So,  some  current  teat  etatlona  are  1050'e-level  dlacrete  logic.  They 
are  unreliable  thlnga,  but  the  premise  on  which  the  training  of  those  people 
la  baaed  la  that  the  test  atatlon  and  the  book  of  tnetructtons  will  allow 
them  to  fix  the  boxea  In  the  pianea. 

When  the  teat  atatlon  works,  that  Idea  works  very  well.  When  the  test 
etatlon  falle,  however,  the  Idea  fade  badty  —  there  are  on  the  order  of  40 
cubic  feet  of  dlacrete  logic  described  by  books  full  of  schematics  that  have 
to  be  ueed  by  somebody  not  well  trained  In  electronics  troubleshooting,  but 
only  In  following  directions. 

That's  the  kind  of  problem  we  have  examined.  We  have  compared  people 
who  were  recently  trained  and  were  doing  well  on  the  Job  with  other  recent 
trainees  who  were  not  doing  as  well. 

If  we  talked  to  firet-ilne  supervisors  and  were  careful  about  how  we 
asked  guest  Ions,  we  could  get  a  reasonably  good  split  of  people  according  to 
their  competence.  We  could  then  do  a  contrastive  analysis,  look  at  the  best 
people,  compare  them  with  the  worst  people,  and  look  at  the  differences. 

We  gave  them  a  battery  of  teats.  Including  some  realistic  trouble¬ 
shooting  tasks  that  were  more  difficult  than  they  characteristically  would 
find  on  the  Job.  But  the  original  split  Is  essentially  asking  the 
supervisor  who  they  think  Is  more  effective. 


It  la  Important  to  note  that  It's  a  face-valid  split  —  people  In  one 
group  can  do  the  work  we'd  like  them  to  do.  which  Is  to  troubleshoot,  and 
the  people  In  the  other  group  are  much  less  able  to  do  that.  We  have  a 
reasonably  good  ability  to  split  the  guys  who  are  moving  along  quickly  from 
the  guys  who  are  moving  along  less  quickly. 

Now,  It  could  be  that  there's  eome  quality  that  eome  people  come  with 
and  other  people  don't,  and  the  people  who  have  that  quality  learn  on  the 
Job  more  quickly  than  other  people,  and  all  that  we're  doing  le  placing 
people  at  different  points  In  the  course  of  learning  on  the  Job.  Even  if 
that's  the  case,  It's  useful  to  know  what  It  Is  that  people  are  slow  at 
learning,  because  that's  a  useful  focus  point  for  more  specific  Instruction. 

What  do  we  know  about  the  more  successful  people?  The  hlgh-ski i l 
people  are  better  at  troubleshooting.  They  are  more  likely  to  find  out 
what’a  wrong  with  the  things  that  they're  trying  to  fix.  They  know  specific 
things  to  do  and  strategies  to  follow  that  are  useful  for  the  purpose  of 
troubleshooting  this  particular  equipment. 

We  can  also  ask  such  questions  as,  "Are  they  more  planful?"  "Are  they 
more  systematic?"  "Do  they  have  dlvlde-and-conquer  strategies?"  There  sre 
no  differences  In  genersl  strategic  capability  per  se.  The  hlgh-skl l l  group 
knows  more  about  the  systems  that  they  work  with  and  the  specific  components 
that  play  a  role  In  troubleshooting  decisions.  They  know  how  those 
components  work.  They  know  how  they  relate  to  the  system  as  a  whole. 
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They  aren't  any  better  at  telling  you  how  a  resistor  works.  They 
don't  have  specific  electronics  knowledge,  and  again,  that's  not  bad.  These 
people  are  doing  their  Job  —  things  are  going  moderately  well.  Some  of 
them  are  doing  their  job  quite  well,  and  they  still  don't  show  a  great 
knowledge  of  electrical  concepts. 

And  finally.  If  we  look  at  the  simplest  level  of  procedures,  we  didn't 
find  sny  difference  In  their  accuracy  In  doing  things,  like  checking  to  see 
whether  a  particular  aMter  reading  fell  within  a  space  given. 

There  were  some  more  complicated  differences  of  a  similar  character. 

For  example.  If  we  look  at  how  reliable  our  people  are  at  making  meter 
readings,  they're  all  about  the  same.  If  we  ask  the  question,  "Old  they 
make  s»eter  readings  when  they  should  have,"  then  we  find  some  Interesting 
differences. 

Particularly,  we  find  that  when  resistance  needs  to  be  measured  In 
order  to  diagnose,  they  were  all  about  equally  good,  and  they  all  tend  to  do 
It.  However,  when  voltage  should  be  measured,  the  less  skilled  group  was 
much  less  likely  to  take  the  measurement  at  the  appropriate  time.  Measuring 
voltage  requires  deeper  understanding  than  measuring  resistance. 

Our  diagnosis  measures  are  measures  of  the  performance  which  they  have 
been  hired  to  do  on  the  Job.  When  we  say  that  they  are  better  at  diagnosis, 
that  Is  their  Job. 

We  split  on  supervisory  ratings,  but  the  truth  of  the  matter  Is  that 
you  always  get  curious  and  you  go  back  and  ask,  "What  If  I  spilt  them  on  the 
basis  of  their  performance  In  the  realistic  troubleshooting  task,"  and  you 
get  much  the  same  kind  of  result. 

The  difference  between  the  two  groups  Is  that  the  doma I n-spec I f I c 
quality  of  the  strategy  varies. 

For  example,  suppose  I  have  a  computer  In  front  of  me  that  has  ten 
boards  In  It,  and  somewhere  else  In  the  shop  there's  a  computer  that  works 
and  also  has  those  ten  boards  In  It.  If  I  were  systematic  In  my  approach  to 
fixing  the  broken  computer,  I  would  take  out  board  number  one,  put  a  good 
board  In,  and  aee  If  that  does  It.  If  that  doesn't  work,  go  to  board  number 
two,  and  eo  on.  That's  a  perfectly  systematic  strategy,  but  It  falls  to 
take  account  of  the  fact  that  the  computer  Is  more  than  the  sum  of  those 
boards. 

Further,  It  falls  to  take  account  of  the  possible  Interactions.  If 
board  number  ten  la  bad,  and  Is  bad  In  the  way  that  will  cause  board  number 
one  to  burn  out,  I  can  play  the  swapping  game  a  long  time.  I  once  saw  a 
technician  replace  the  same  board  seven  times  and  he  smoked  each  one. 

People  can  behave  very  systematically  and  Just  not  be  doing  the  right  thing. 

If  you  look  at  the  question  the  way  we  posed  It,  I  think  you  would  say 
that  we  found  higher  skill  people  have  more  strong  methods  that  would 
generalize.  It  wasn't  our  Intention  to  make  that  cut,  but  If  you  look  at 
what  we're  calling  strong  methods,  there  are  things  like  tracing  outs, 
following  schematics,  and  making  certain  kinds  of  measurements  with  the 
meter . 
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These  functions  are  not,  In  fact,  specifics  for  one  particular  test 
station,  in  fact,  It's  the  leas>skl l led  person  that  says,  "Gee,  the  only 
t late  I  saw  one  of  those  fall.  It  was  because  board  number  43  was  bad.  I'm 
going  to  try  board  number  43." 

If  you  look  at  what  the  less-skilled  persons  were  taught  compared  to 
what  the  hlgh-skl l l  people  seem  to  be  better  at,  a  certain  kind  of 
generalization  can  be  made.  In  the  declarative  knowledge  arena.  If  you're 
looking  at  factual  knowledge  concepts  of  electronics,  they  were  taught 
extremely  abstract  concepts  at  the  very  beginning  of  their  careers. 

Bhat  they  don't  get  at  the  beginning  le  the  more  concrete  stuff  that 
has  the  character  of  the  chapter  In  any  reference  manual  called  "theory  of 
operation."  If  you're  trying  to  narrow  the  problem  to  one  of  ten  boxes  In 
this  big  machine,  then  knowing  what  those  ten  boxes  do  can  be  very 
Important . 

Knowing  the  function  of  resistors  may  or  may  not  be  as  Important  for 
that  piece  of  the  decision.  One  thing  that  seems  to  be  the  case  Is  that  the 
declarative  knowledge  that’s  provided  le  too  abstract  either  to  foster  easy 
learning  or  to  enable  good  troubleshooting. 

On  the  other  hand,  the  procedural  knowledge  that  le  provided  in 
training  seems  to  be  too  concrete,  and  characteristically  Involves  the 
practice  of  surely  following  the  Instructions  that  are  In  the  technical 
orders. 

It  Is  a  replicable  and  predictable  accident  that  troubleshooting  Jobs 
tend  to  Involve  more  real  problem  solving  than  the  system  designer 
anticipated. 

Now,  let  me  try  to  deal  with  the  second  point.  It  le  Interesting  to 
look  at  what  really  happens  when  we  try  to  build  a  learning  hierarchy  or 
ordered  characterization  of  the  goals  of  Instruct  ion. 

There  are  many  different  viewpoints  one  can  take  on  how  to  split  up  the 
big  connected  fabric  of  knowledge  that  we  want  people  to  develop.  In  the 
case  of  some  simple  electronics,  we  can  categorize  by  which  concepts  are 
being  used;  by  kinds  of  circuits,  parallel  and  aeries;  and  by  kinds  of  laws. 
All  of  those  splits  tend  to  represent  different  viewpoints  on  the  same  basic 
task  —  we  have  a  big  body  of  highly  Interconnected  knowledge  that  we  want 
to  convey  to  somebody. 

How  we  make  those  splits  though,  can  cause  certain  things  to  happen. 
Because  of  the  need  for  accountability  and  the  need  for  modularity  In 
Instruction,  we  spilt  the  pie  Into  some  pieces.  Different  people  might  be 
In  charge  of  teaching  those  different  pieces.  Each  of  those  people  le  going 
to  want  to  verify  that  they  have.  In  fact,  correctly  and  completely  done 
what  they  were  assigned  to  do.  So,  they're  going  to  test  whether  or  not 
they  have  taught  the  piece  we  gave  them  to  teach,  and  they're  going  to 
direct  all  of  their  attention  to  optimizing  performance  In  those  tests. 

For  example,  let’s  say  there  are  three  things  In  particular  that  are 
useful  to  know  In  troubleshooting  these  big  test  stations.  When  we  build 
our  lesrnlng  hierarchy,  as  we  accumulate  subskl I  la  Into  certain  higher  level 
skills,  there’s  always  some  material  left  out.  Some  Information  la  implicit 


and  we  hope  the  student  will  figure  out  on  hie  own  —  some  glue  that  takes 
these  separate  pieces  of  knowledge  and  hooks  them  together  to  wake  them 
ueeable. 

If  I  am  In  charge  of  teaching  Ohm's  Law,  and  I  look  at  all  the  things 
I  know  about  Ohm's  Law,  what  am  I  going  to  give  as  a  test  to  prove  that  I 
really  taught  the  guy  Ohm'e  Law?  What  am  I  going  to  teach  him?  I'm  going 
to  find  the  very  core  of  that  Ohm's  Law  knowledge  —  the  thing  that  looks 
more  like  Ohm's  Law  than  anything  else,  so  that  when  people  get  that  test 
Item  correct,  I  can  smile  and  say,  "I  did  my  part.”  I'm  going  to  focus  my 
attention  on  the  center  of  this  piece  of  the  fabric  of. knowledge  we  want  to 
convey . 

When  we're  finished,  we  have  a  situation  In  which  the  Ohm's  Law  teacher 
taught  the  central  things  about  Ohm's  Law,  and  the  dl vlde-and-conquer 
strategy  teacher  taught  some  of  the  central  things  about  dl vlde-and-conquer 
strategy.  Each  of  them  can  demonstrate  conclusively  with  their  tests  that 
what  they  set  out  to  teach,  they  In  fact  taught.  But  there's  a  big 
assumption  that  the  Inter-relatlonshlpa  between  those  pieces  of  knowledge, 
will  somehow,  be  successfully  acquired  by  the  student. 

What  do  l  do  If  l  can’t  teach  the  student  to  troubleshoot  the  test 
stations,  after  he  has  taken  these  course  modules?  I  complain  about  the 
fact  that  he  doesn't  know  Ohm's  Law  and  sieter  reading,  because  he  can't  use 
them  to  do  what  I  want  him  to  do.  I  send  him  back  to  those  teachers  and 
they  teach  again  the  same  thing  that  they  probably  successfully  taught 
before.  This  Is  Insufficient.  Different  viewpoints  are  needed  on  the 
target  knowledge  for  Instruction,  for  remediation,  and  for  assessing 
transfer . 

If,  for  example,  we  want  to  have  somebody  know  those  aspects  of  Ohm’s 
Law  that  are  critical  to  troubleshooting,  we  need  to  Identify  and  understand 
those  aspects.  When  we  find  that  people  aren’t  learning  to  troubleshoot,  we 
need  to  be  sophisticated  enough  to  understand  that  this  Is  not  a  simple 
question  of  needing  two  weeks  Instead  of  one  week's  study  of  Ohm's  Law. 
Rather,  there  Is  some  other  material  that  needs  to  be  added  that  wasn't  the 
target  of  Instruction  In  the  first  place. 

Let  me  present  an  example  to  Illustrate  my  third  point.  The  example  Is 
from  Herman  Hartel,  who  looks  at  children  In  the  top  end  of  the  gymnasium  in 
Germany.  They  get  several  years  of  physics  and  you  can  convince  yourself 
that  these  students  really  understand  the  basic  concepts,  like  Klrchhoff's 
Law,  that  help  up  apply  some  of  the  things  that  we  know  about  electricity. 

They  can  pass  any  sort  of  standard  test  that  you  could  pose  to  those 
people.  Then  Hartel  gives  them  a  problem  using  real  devices  —  wires, 
bulbs,  and  batteries.  He  says,  "Look,  here  Is  this  battery  and  the  wire 
goes  from  here  to  this  lamp  and  then  over  the  next  lamp,  and  then  it  goes 
back  there,  and  this  lamp  lights  up  and  this  lamp  doesn’t.  What's  going 
on?" 

And  what  do  the  students  say?  They  say  something  like,  "Well,  the 
electricity  gets  through  this  bulb  and  then  It  stops  here  because  this  lamp 
Is  broken." 
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Tested  In  any  conventional  way.  they  will  behave  ae  If  they  know  all 
the  lawe  and  principled  of  electricity  that  aaye  "this  e imply  cannot 
happen . “ 

The  point  la  that  we  would  hope  these  really  bright  etudente  would 
simply  realize  that  thle  le  a  aerlee  circuit,  and  If  the  circuit  la  closed, 
which  le  the  only  way  thle  lamp  can  light,  the  current  has  to  be  passing 
through  that  lamp.  Whatever  solution  I  come  up  with,  It  has  to  have  that 
aspect.  Inetead,  they  retreat  to  a  fundamental  wlsconcept Ion  —  that 
electricity  le  the  stuff  that  you  push  through  wires,  like  electricity 
hosee.  and  It's  quite  possible  for  It  to  pase  through  this  lamp  and  light  It 
and  then  stop  at  the  second  light. 

The  point  l  want  to  make  Is  that  Instruction  In  areas  where  there  are 
theee  fundamental  ail sconcept Ions  has  the  character  of  war.  It's  a  war 
against  aiiseoncept Ion,  and  when  you  fight  a  war  there  are  certain  basic 
principles  that  need  to  be  kept  In  ailnd.  One  of  those  principles  la  that  a 
particular  aswunt  of  force  will  produce  a  victory  only  If  used  properly.  At 
tlaies  you  need  to  use  overwhelming  force  in  order  to  absolutely  demoralize 
the  enemy. 

In  this  sense  misconceptions  that  persevere  represent  fundamental 
conflicts  with  the  skills  that  we  want  people  to  acquire,  and  they  have  to 
be  shattered.  That  la  the  classic  approach. 

There  are  other  things  we  can  do  that  work  better  If  we  don't  have  the 
resources  to  try  the  overwhelming  force  approach.  For  example,  one  approach 
le  the  ancient  tactic  of  building  a  aeries  of  garrisons.  We  have  to  stake 
out  pieces  of  territory  In  order  to  retain  control.  That  Is  a  relatively 
expensive  strategy. 

In  the  case  of  conducting  a  war  on  misconception,  that  would  swan 
having  to  provide  enough  pockets  of  knowledge  to  sufficiently  cover  the 
domain  In  question.  In  thle  way,  we  can  count  on  the  misconception  being 
unable  to  reassert  Its  control. 

The  third  strategy  la  to  win  the  hearts  and  minds  of  the  people.  You 
can  provide  some  Infrastructure  that's  there  on  your  aide  even  when  you  are 
absent.  We  do  that  when  we  teach  certain  kinds  of  metacognlt Ive  skills  — 
when  we  convince  people  that  certain  constraints  must  hold  In  a  domain  and 
that  they  have  to  reason  their  way  around  the  apparent  contradictions  to 
those  constraints. 

I  would  assert  that  when  we  think  about  transfer,  we  oust  keep  In  mind 
the  possibility  that  transfer  Is  often  to  tasks  to  which  the  person  does 
have  relevant  knowledge,  but  some  of  that  relevant  knowledge  Is  wrong.  In 
electronics,  for  Instance,  when  we  deal  with  certain  mechanical  kinds  of 
situations,  people  are  equipped  with  a  supply  of  successful  principles  that 
don't  general ize. 

Those  principles  have  been  reinforced  in  the  average  recruit  by  18 
years  of  experience  In  the  world.  Even  If  we  taught  a  lot  of  the  right 
things,  the  tranefer  situation  —  unleee  It's  been  planned  for  —  will 
simply  fall  Into  one  of  those  holes,  which  brings  about  the  old 
ml sconcept Ions. 


We  can  work  with  an  expert  ahead  of  tlwe  and  00  Into  a  training 
situation  knowing  In  advance  almost  every  reaponse  —  both  correct  and 
Incorrect  —  that'e  going  to  be  wade  in  a  very  complex  troubleshooting  task. 
That  means  we  can  get  to  the  point  of  knowing  where  to  head  off 
Inappropriate  experiences. 

A  final  point  I  would  like  to  make  la  that  I  think  psychologists 
grossly  underestimate  the  extent  to  which  they  need  to  be  knowledgeable  In 
the  domain  In  which  they  are  working.  If  you  look  at  the  first  year  or  two 
of  our  work.  It  suffered  greatly  from  being  good  psychology  and  superficial 
electronics.  It  was  only  after  everyone  on  the  project  attended  regular 
electronics  seminars  that  we  began  doing  really  useful  work. 
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I  want  to  raise  some  Issues  today  related  to  transfer  of  training  that 
have  to  do  with  applied  performance  testing.  They  are  in  the  areas  of 
fidelity  of  measurement  and  the  process  by  which  we  define  what  It  Is  that 
we  want  to  aieasure. 

I  have  spent  the  past  ten  years  at  the  Navy  Personnel  Research  and 
Development  Center  doing  research  Involving  applied  performance  testing. 
Initially  my  work  Involved  building  diagnostic  testing  and  shipboard 
training  packages.  This  was  followed  by  work  on  proficiency  assessment  to 
provide  feedback  to  the  training  community. 

About  three  years  ago,  I  switched  from  working  In  the  training  area  to 
the  manpower  and  personnel  area.  Thus,  my  comments  today  will  reflect  more 
of  the  viewpoint  that  would  be  taken  by  a  personnel  psychologist  than  a 
training  special  1st. 

The  first  Issue  I  want  to  raise  concerns  the  fidelity  of  performance 
measurement.  Applied  performance  testing,  conducted  at  the  appropriate 
measurement  levels,  may  be  the  best  way  to  measure  and  ensure  transfer  of 
training.  By  applied  performance  testing,  I  mean  testing  the  Individual  on 
the  tasks  that  they  actually  do  on  the  Job  using  the  equipment  that  they  use 
on  the  Job. 

This  Is  an  extremely  high  fidelity  type  of  measure  —  the  type  of 
measure  that  almost  everyone  would  agree  that  you  should  use  to  assess 
whether  there  Is  any  transfer  of  training  to  the  Job.  Unfortunately,  this 
type  of  high  fidelity  test  Is  extremely  expensive. 

One  of  the  major  contributions  of  work  now  being  done  In  applied 
performance  testing  will  be  the  development  of  a  technology  for  building  and 
evaluating  hands-on  Job  performance  teats  —  and  perhaps  other  measures  that 
can  be  used  In  place  of  these  Job  sample  types  of  tests. 

We  have  the  complete  technology  to  build  paper-and-penc l I  tests,  and  we 
•Iso  have  a  solid  foundation  for  the  technology  to  build  criterion- 
referenced  teats,  but  we  don’t  have  the  same  technology  for  constructing 
hands-on  performance  tests. 

This  lack  of  technology  has  Implications  for  transfer  of  training  work 
as  well  as  for  the  type  of  validation  work  I  do  now.  For  example,  we  don't 
have  any  good  technical  guidance  for  selecting  transfer  tasks  to  properly 
assess  transfer  of  training  or  for  selecting  tasks  for  a  test  to  validate 


••l*ctlon  Instruments.  Clearly,  performance  testing  Is  an  area  In  need  of 
additional  research. 

It  •seme  to  me  that  It  la  Important  In  pursuing  future  research  to 
develop  some  hlgh-f Ids  I Ity  but  low-coat  performance  awasures,  whether  our 
purpose  Is  to  assess  transfer  of  training  or  validate  predictor  tests. 

It  has  already  been  mentioned  that  hands-on  perfornance  testing  Is 
expensive.  It  Is  expensive  to  build  the  test,  and  then  very  expensive  to 
administer  the  test  In  a  one-on-one  situation.  In  addition,  we  take  up  the 
resources  of  the  community  In  which  we  work  —  In  equipment  use,  for 
example.  If  we  use  a  less  expensive,  lower  fidelity  test,  however,  we  do 
not  know  what  effect  the  use  of  such  a  test  will  have  on  performance 
measurement . 

To  illustrate  an  approach  that  I  feel  will  contribute  to  the  needed 
technology  of  performance  testing,  I  would  like  to  present  some  Information 
about  a  validation  project  with  which  I  am  Involved.  There  are  about  05 
different  Jobs  In  the  Navy  to  which  we  would  like  to  be  able  to 
differentially  assign  recruits  to  Mxlmlze  job  perforates. 

Neither  the  Navy  nor  any  of  the  other  Services  can  afford  to  develop 
and  administer  hsnds-on  performance  tests  for  all  Jobs  as  we  routinely 
validate  and  update  our  personnel  classification  and  assignment  procedures. 
Therefore,  for  a  limited  number  of  Jobs,  a  Joint-Service  working  group 
decided  to  use  hands-on  tests  as  "benchmark*  measures  against  which  some 
less  expensive,  store  easily  administered  substitute  awasures  will  be 
compared  In  large  field  experiments. 

In  the  Navy,  we  are  looking  at  two  types  of  substitute  measures.  One 
Involves  simulation;  either  a  paper-and-penc 1 1  test  or  a  computer-based 
simulation.  The  second  substitute  measure  Is  a  set  of  behavioral ly  anchored 
rating  scales.  We  plan  experiments  to  look  at  the  relationships  among  the 
different  types  of  measures  and  between  these  measures  and  the  major 
•election  Instrument  used,  the  Armed  Services  Vocational  Aptitude  Battery  or 
ASVAB . 

The  other  Issue  that  I  want  to  rales  has  to  do  with  defining  what  It  is 
that  we  are  measuring. 

In  the  case  of  the  Joint-Service  working  group  that  I  Just  mentioned, 
we  found  It  easy  to  talk  about  relating  the  ASVAB  to  Job  performance,  but 
found  it  difficult  to  get  agreement  on  what  constitutes  a  comprehensive 
measure  of  Job  performance.  When  we  discussed  what  kinds  of  measures  we 
were  going  to  use.  there  was  concern  that  we  were  really  not  measuring  job 
performance,  or  that  we  were  aMasuring  too  much  of  It  or  not  enough  of  It. 

Bosm  Induatr la  I/organ l zat Iona  I  psychologists  suggest,  for  example,  that 
we  must  look  not  only  at  whether  an  Individual  fiin  do  the  Job,  but  does  he 
or  ehe  do  the  Job?  Are  the  Individual e  really  there  on  the  job?  Are  they 
absent,  do  they  use  drugs,  are  they  motivated?  That  is.  do  they  engage  in 
"down-time"  behaviors?  What  about  the  observance  of  safety  precautions?  Do 
they  engage  In  "hazardous"  behaviors? 

All  of  these  things  make  up  general  performance  on  the  Job.  But  there 
continued  to  be  confusion  In  the  working  group  about  what  aspect  or  aspects 


of  Job  performance  we  should  be  measuring.  The  adequacy  of  any  performance 
aieasure  will  depend  on  the  purpose  of  the  measurement,  the  measurement 
method  used,  and  what  Is  aaini  by  Job  performance.  We  finally  decided  to 
take  a  narrow  focus  and  look  only  at  technical  proficiency.  That  Is,  we  are 
going  to  test  only  the  “can  do*  aspect  of  Job  performance  using  tasks  of  a 
technical  nature  done  on  the  Job. 

To  put  this  decision  In  perspective,  Kevin  Murphy  from  Colorado  State 
University  and  several  of  us  at  the  Navy  Personnel  Research  and  Development 
Center  tried  to  lay  out  what  we  awant  when  we  talked  about  Job  performance 
and  we  came  up  with  a  set  of  performance  levels.  These  levels  are  most 
easily  thought  of  as  a  type  of  flow  chart,  because  It  follows  an  Individual 
from  the  time  that  he  or  she  enters  the  Service,  to  the  point  of  post¬ 
training  Job  performance.  This  Is  shown  as  five  levels: 

I.  Effectiveness  In  a  position 

II.  Behavioral  components  of  effectiveness 

III.  interpersonal  relations 

IV.  Job  or  task  proficiency 

V.  Skill  and  knowledge  components  of  proficiency 

The  flow  Is  from  underlying  skills  and  knowledge  at  Level  V  to 
effectiveness  In  a  position  at  Level  I.  Training  occurs  at  Level  v,  but 
practice  and  experience  In  the  job  environment  Is  necessary  to  achieve  task 
proficiency  at  level  IV.  Task  proficiency  and  the  other  behaviors 
represented  at  Levels  III  and  II,  such  as  the  down-time  and  hazardous 
behaviors  aientloned  earlier  and  Interpersonal  skills,  contribute  to 
effectiveness  In  a  position.  Effectiveness  determines  the  value  of  the 
Individual  to  the  organization. 

l  believe  that  this  concept  of  Job  performance  has  application  to  the 
assessment  of  transfer  of  training.  Ultimately  we  want  to  bring  trainees 
into  the  Job  and  have  them  be  effective.  That  Is,  we  want  training  to 
transfer  to  the  Job.  However,  we  cannot  determine  how  aiuch  transfer  of 
training  occurs.  If  any,  unless  we  can  specify  the  Job  performance  universe 
and  relate  performance  measures  to  it. 

It  Is  not  until  we  Identify  the  concept  of  achievement  In  different 
performance  areas,  such  as  those  represented  In  the  levels  of  performance  1 
have  been  talking  about,  that  we  will  know  what  It  Is  that  we  ought  to 
measure  to  assess  transfer  of  training.  The  main  advantage  of  having  a  well 
defined  concept  of  Job  performance  Is  that  It  will  allow  us  to  decide  what 
the  transfer  tasks  ought  to  be  and  how  to  set  up  the  situations  to  measure 
transfer  of  training. 

Another  advantage  of  laying  out  a  concept  of  Job  performance  using 
different  performance  levels  Is  that  we  can  see  that  we  have  to  measure  at 
different  levels  for  different  purposes. 

We  made  a  conscious  decision  In  the  Joint-Service  project  I  described 
to  look  at  task  proficiency  because  performance  at  that  level  Is  most  likely 
related  to  the  kinds  of  abilities  that  we  are  measuring  In  the  current  ASVAB 
or  in  some  of  the  new  ability  predictor  tests  that  we  are  also 
Invest Igat Ing. 
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Likewise,  we  «ay  have  to  make  decisions  to  Masure  at  different 
performance  levels  to:  (1)  diagnose  learning  difficulties,  document 
achievement,  and  ensure  transfer  of  training  within  a  course  of  instruction, 

(2)  measure  transfer  of  training  to  proficiency  In  a  specific  Job  or  task, 

(3)  ensure  that  transfer  of  training  contributes  to  global  Job  behaviors  or 
effect Iveness  In  a  specific  position,  or  (4)  provide  quality  control 
InforMtlon  or  feedback  to  the  training  program. 

The  next  Item  l  want  to  talk  about  concerns  providing  more  detail  at 
the  different  levels  of  perforMnce.  An  overall  concept  like  Job 
perforMnce  has  to  pontaln  detailed  perforMnce  constructs.  Analytic  tools 
are  needed  to  specify  the  cognitive  processes  and  memory  structures  that 
make  up  these  performance  constructs. 

If  we  can  analyze  Job  performance  using  a  cognitive  task  ana'ysls 
procedure,  we  can  determine  what  should  be  taught  to  Individuals  to  enable 
them  to  become  proficient.  Performance  constructs  can  help  point  out 
discrepancies  between  what  Is  being  trained  and  what  needs  to  be  trained, 
and  even  give  us  Ideas  on  the  appropriate  sequencing  of  training. 

Performance  constructs  also  can  provide  guidance  on  what  to  test  and  how  to 
test  It  to  Met  various  training  Masurement  purposes. 

Although  It  may  be  possible  to  achieve  Job  and  task  proficiency  solely 
through  "on-the-job"  training,  It  is  usually  Impractical.  This  Mans  we 
need  s  mix  of  classroom  and  on-the-job  training.  Specifying  a  performance 
construct  and  taking  Into  account  the  resources  available  In  the  training 
and  Job  enviroranents,  may  allow  us  to  Mke  eoM  decisions  on  the  best  mix  of 
training  enviroranents. 

What  I  am  really  proposing  la  that  m  take  a  "systems  approach"  to 
problems  common  to  both  validation  and  transfer  of  training  work  —  for  that 
we  have  to  apply  the  personnel  psychologist *s  notion  of  construct  validation 
when  defining  and  Masurlng  Job  performance,  and  we  have  to  capitalize  on 
the  training  specialist's  capability  to  analyze  the  underlying  skills  and 
abilities  that  are  related  to  Job  perforMnce. 

In  summary,  m  need:  (1)  studies  of  perforMnce  test  fidelity  that  use 
different  MasureMnt  Mthods  and  tie  these  Mthods  to  different  performance 
levels,  and  (2)  a  cognitive  task  analysis  procedure  that  can  serve  both 
selection  snd  training  research. 

There  already  have  been  calls  for  a  construct  validation  approach  to 
cr Iter  Ion-referenced  testing  and  scholastic  aehlevsMnt.  According  to  such 
an  approach,  defining  the  perforMnce  universe  and  then  specifying  a 
perforMnce  construct,  that  Is,  the  perforMnce  domain,  Is  an  Important  step 
In  Masurlng  transfer  of  training  to  the  Job.  Low-cost,  hlgh-f Idei Ity 
Masures  of  performance  are  the  key  to  success  In  this  enterprise,  and  a 
cognitive  task  analysis  procedure  can  help  us  build  fidelity  Into  our 
Masures. 

A  cognitive  task  analysis  procedure  also  can  provide  the  thread  that 
ties  personnel  selection  to  Job  perforMnce  via  training.  Personnel 
selection  Influences  training  by  Mtehlng  predictors  to  the  Mthod  of 
Instruction  and/or  course  content;  It  Influences  Job  performance  by  matching 
predictors  to  the  Job.  The  same  underlying  cognitive  processes  and 
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structures  that  asks  up  the  Job  performance  construct  should  be  represented 
In  the  predictors,  training  content,  and  the  Job  tasks. 

The  approach  and  procedures  that  I  have  advocated  today  Imply  a  general 
need  for  a  theory  of  human  performance  that  will  encompass  both  transfer  of 
training  and  performance  prediction. 
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l  want  to  talk  about  building  what  I  will  refer  to  as  automatic 
processing  component  skills.  I  will  begin  with  some  theory,  and  then  apply 
It  to  transfer  of  training.  I  will  describe  the  distinction  between 
automatic  and  controlled  processing.  The  Interpretation  of  these 
differences  Is  useful  for  understanding  how  skill  Is  acquired  and  help 
clarify  some  misperceptions  of  the  role  of  training,  and  how  training 
translates  into  performance. 

I'll  also  talk  about  some  empirical  changes  with  practice,  In 
particular  the  Increases  In  processing  speed  and  reduced  workload  that  are 
necessary  for  high  performance  skills.  Then  I'll  talk  about  the 
characteristic  of  Increased  reliability.  You  may.  In  a  training  context, 
want  to  train  somebody  not  only  so  they  can  perform  the  task,  but  perform  it 
when  they  didn't  have  enough  sleep  the  night  before,  or  have  a  problem  with 
alcohol,  or  haven't  refreshed  the  training,  or  haven't  experienced  the 
training  In  six  months  of  the  task.  I  will  also  talk  about  the 
characteristic  of  negative  transfer,  when  training  can  Influence  processing 
so  you  don't  do  something,  or  you  do  something  that  you  didn't  Intend  to  do. 

Finally  I  will  go  Into  some  guidelines  for  developing  automatic 
processes,  and  talk  a  little  bit  about  the  challenge  of  skill  training. 

One  character  1st lc  of  training  that  evolves  from  my  work  Is,  there's  no  such 
thing  as  a  free  lunch.  If  you  want  high  performance  you  are  going  to  have  to 
spend  a  substantial  amount  of  time  In  training,  and  developing  the  tasks. 
I'll  give  you  a  brief  description  of  training  guidelines  from  this 
perspective.  I'll  take  some  examples  of  two  training  systems  that  we're 
working  with  at  the  moment,  one  for  training  electronic  troubleshooting,  and 
the  other  for  air  Intercept  control. 

I  view  human  performance  as  being  the  result  of  two  Qualitatively 
different  processes.  The  first  la  controlled  processing,  which  Is  typical 
of  novel  situations.  It  la  illustrated  by  being  slow,  serial,  effortful  — 
a  conscious  processing  of  very  limited  capacity  and  under  direct  subject 
control.  Most  declarative  knowledge  processing  Involves  controlled 
processing. 

The  second  mode  of  processing,  which  Is  more  typical  of  the  highly 
skilled  practitioner  with  years  of  training  on  the  Job,  Is  automatic 
processing.  This  Is  typical  of  situations  where  you  do  the  same  thing  time 
after  time  after  time.  It's  character ized  as  fast,  parallel,  fairly 
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•f fort Ittt,  and  not  Malted  by  short-term  aemory  oompar Isons.  I  will  claim 
that  a  major  part  of  training,  at  least  In  the  area  of  skill  development.  Is 
the  building  up  of  these  automatic  performance  skills. 

To  give  you  an  appreciation  of  the  magnitude  of  these  effects,  I  will 
give  you  some  examples  from  a  basic  research  paradigm.  The  differences 
between  these  types  of  processing  are  or  can  be  two  orders  of  magnitude.  A 
frequently  used  paradigm  Is  a  search  task.  In  such  a  task  you  might  ‘Push 
button  1  If  you  see  the  letter  X  or  L,"  then  you  would  see  a  set  of  stimuli, 
and  you  would  make  a  response.  You  see  a  aeries  of  presentations  of  single 
letters,  where  you  have  to  make  a  simple  reeponse  to  a  email  set  of  letters. 

An  Important  distinction  In  this  work  between  the  automatic  and 
controlled  processing  Is  the  type  of  sapping  of  the  stimuli  and  the  task. 

If  It's  a  consistent  mapping,  the  response  Is  the  same  every  time  you 
perform  the  task  (e.g.,  dialing  your  home  phone  number).  In  a  varied 
mapping,  the  stimulus  and  response  change  from  trial  to  trial  (e.g.,  dialing 
random  phone  numbers) . 

In  a  consistent  search  task,  you  would  be  told  In  the  trial  "Push  a 
button  If  you  see  X  or  L."  L  occurs  and  you  push  button  1;  If  a  K  occurs, 
you  push  a  2.  Later,  you  see  0  or  Q.  D  occurs,  and  you  push  button  1.  The 
critical  thing  Is,  you  never  make  a  different  response  to  the  particular 
stimulus.  In  a  varied  search  task  you  might  search  for  a  P  or  L  on  one 
trial  and  push  button  1  for  an  L  on  trial  1.  However,  If  on  the  next  trial, 
while  searching  for  an  X  or  G,  an  L  occurred,  you  would  respond  2.  The 
response  to  the  L  would  be  var led  (l.e.,  first  1  then  2)  across  trials.  In 
this  varied  condition  only  controlled  processing  Is  expected. 

In  the  normal  work  environment  most  tasks  are  consistent  but  varied 
tasks  occur  and  do  cause  troubles.  The  pressing  of  keys  In  typing,  starting 
of  an  aircraft  engine,  or  driving  a  car  on  an  empty  road  are  consistent 
tasks.  Human  computer  Interactions  provide  many  examples  of  varied  tasks. 
For  example,  different  programmers  use  different  keys  to  leave  a  program 
(e.g.,  press  Esc,  a  function  key,  type  "bye"  or  "quit").  The  Internal 
stimulus  of  "exit  the  program"  la  mapped  to  different  responses.  Such 
designs  result  In  frequent  errors  and  Increased  workload. 

In  the  design  of  complex  eystems,  there’s  a  tendency  for  engineers  not 
to  worry  about  the  consistency  of  the  commande  and  the  motor  responses 
across  all  the  methods  of  operation  of  that  system.  This  Is  a  varied 
mapping  situation.  The  sequence  that  lets  me  do  one  thing  when  I'm  using 
the  editor  will  disconnect  me  If  I'm  using  the  modem.  I  have  to  keep  In 
mind  the  goal,  because  the  correct  response  is  not  consistent. 

Let  me  begin  with  a  basic  effect.  We've  looked  at  these  two  types  of 
processing  —  what  the  effect  la  of  training  In  the  controlled  mode  where 
you  have  this  varied  mapping,  versus  the  consistent  mapping  where  you 
develop  automatic  processing. 

Using  such  letter  search  tasks,  we  have  found  no  Improvement  In 
detection  accuracy  In  the  varied  mapping,  while  finding  quite  substantial 
Improvement  In  performance  In  the  consistent  mapping.  The  take-home  me99age 
Is  that  practice  Improves  primarily  on  consistent  tasks  (Schneider,  1285, 
Schneider  A  Flak,  1982a). 
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The  statement  that  practice  makes  perfect  la  fai8e.  If  you  are 
practicing  one  of  these  varied  mapping  tasks,  which  we  have  done  for  up  to 
six  months,  performance  doesn't  change  much  after  the  first  half-hour. 
Extended  training  time  In  dealing  with  those  varied  components  doesn't 
result  In  Improvement,  and  Is  a  waste  In  terms  of  your  processing 
improvement . 

The  second  characteristic  that  we  typically  want  of  a  skilled 
practitioner  la  rapid  processing  of  a  task.  I  want  to  give  you  an  example 
of  a  search  task,  and  then  I’ll  talk  about  the  data.  The  Important 
character  1st Ic  Is,  automatic  processes  typically  become  much  more  rapid. 

Let  me  give  you  a  flavor  of  what  happens  In  a  trial. 

On  a  trial,  a  subject  searches  for  words  from  one  to  four  categories 
(e.g.,  FURNITURE,  CITY,  TREE,  JOB).  The  subject  sees  a  display  containing 
two  words  (e.g.,  YELLOW,  ELM)  and  responds  If  there  Is  a  match  (e.g.,  TREE 
to  ELM). 

What  we  find  In  such  tasks  Is  performance  speed  Is  a 
I  Inear l y- Increasing  function  of  the  number  of  memory  comparisons  In  the 
varied  mapping  condition  (e.g.,  searching  for  one  category  requires  .5 
seconds,  two  categories,  .7,  three  categories,  .9).  It  takes  about  200 
milliseconds  to  make  a  semantic  comparison,  whether  this  Is  on  day  1  or  on 
day  25  of  the  task  (Fisk  &  Schneider,  1983).  However,  If  you  are  always 
looking  for  the  same  set  of  elements,  and  responding  to  them  the  same  way, 
reaction  time  doesn't  Increase  (l.e.,  search  time  for  one  to  four  categories 
Is  .5  seconds). 

Figure  1  shows  the  difference  In  memory  comparison  time  between 
controlled  and  automatic  processing.  Comparing  the  slopes  using  the 
negative  trial  responses  we  find  that  the  controlled  processing  Is  nearly  a 
hundred  times  slower  than  automatic  processing  of  the  category  Judgment 
task. 


The  cost  of  giving  somebody  another  stimulus  to  look  for,  as  long  as  it 
Is  responded  to  consistently.  Is  negligible.  In  the  case  of  the  automatic 
mode,  the  marginal  Increase  was  2  milliseconds  (Fisk  1  Schneider,  1983). 
Through  consistent  practice  there  is  a  shift  from  serial  to  parallel 
processing.  You  can  do  several  things  at  once,  and  we  have  approximately  a 
two-order  of  magnitude  speed-up  In  processing  time.  In  many  situations, 
particularly  problem-solving  or  diagnosis,  you  want  to  look  through  many 
alternatives  very  quickly. 

I  should  mention,  any  time  I'm  talking  about  the  automatic  mode,  I  am 
typically  looking  at  somebody  who  has  had  In  excess  of  2,000  trials  of 
performing  the  task.  Substantial  automat Iclty  can  develop  by  about  200 
trials.  Automat Iclty  doesn't  come  cheap,  but  It  can  be  very  beneficial  In 
terms  of  the  nature  of  the  processing  speed-up. 

This  processing  speed-up  Issue  is  a  general,  lawful  relationship  in 
psychology,  referred  to  as  the  power  log  of  practice  (see  Newell  t 
Rosenbloom,  1981)  —  the  log  of  the  reaction  time  Is  predicted  by  the  log  of 
the  number  of  trials.  This  holds  for  clgar-rol I Ing,  playing  bridge,  and 
geometry  proofs.  It's  a  very  general  phenomenon.  General  enough  that  l 
think.  If  you're  trying  to  do  some  evaluation  of  a  training  program,  you  can 
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Figure  1.  Representation  of  oarrparisen  slopes  for  stimuli  processed  using 
controlled  processing  versus  automatic  processing.  (Adapted  from  Fisk  & 
Schneider,  1983) 
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Initially  get  an  eatlmate  of  this  slope  and  make  a  projection  that  will 
allow  you  to  estimate  the  number  of  trials  needed  to  Increase  performance  by 
20  percent. 

The  take-home  message  from  the  standpoint  of  training  design  Is  that 
Investments  In  training  pay  off  In  processing  speed.  If  the  number  of 
practice  trials  Is  Increased  from  10  to  100,  processing  efficiency  will 
Increase  66  percent  (see  Anderson  In  Newell  6  Roaenbloom,  1081).  With  this 
kind  of  functional  relationship,  we  can  start  building  prediction  Into 
training  programs. 

We  have  talked  a  little  about  a  toolkit  for  s  training  designer.  I 
think  these  types  of  lawful  relationships  can  be  Implemented  as  a  subroutine 
In  an  expert  system  or  programming  package.  Then  you  could  provide  an 
engineer  the  most  effective  tradeoff  In  training  options,  to  give  you  the 
best  performance. 

Training  speedup  Is  one  aspect.  An  (siportant  aspect  of  Improvements  in 
practice  Is  reduced  workload.  This  has  obvious  application  to  military 
settings,  where  In  the  last  30  seconds  of  releasing  a  weapon  there  Is  a 
tremendous  workload.  However,  I  believe  It  also  has  applications  to  much 
store  mundane  tasks,  and  In  particular  I  want  to  look  at  electronic 
troubleshooting.  In  this  controlled  mode  In  a  task  using  declarative 
knowledge,  the  task  Is  learned  quickly.  I  want  to  give  you  a  short 
electronics  course  here.  A  NANO  gate,  which  IS  NOT  AND.  has  an  output  of 
zero.  If  all  the  Inputs  are  one,  and  output  one  the  rest  of  the  time.  What 
Is  the  output  If  the  Inputs  are  “10",  “11",  "00“?  It  turns  out,  to  learn 
the  correct  outputs  takes  about  SO  trials  of  training.  By  the  end  of  a 
training  course  In  the  classroom  a  student  probably  gets  about  that  many 
trials  to  practice.  This  Is  the  performance  level  typically  assessed  with 
paper  and  pencil  tests. 

However,  the  electronic  troubleshooter  In  the  field  will  encounter  a 
test  point  A  which  has  a  value  of  one,  K  which  has  a  value  of  zero,  J  which 
has  a  value  of  one,  and  T  which  has  a  value  of  zero.  The  person  must 
remember  the  test  points  rule  for  the  gate. 

Let 'e  try.  What's  the  output  If  the  Input  Is  "KO",  "A7a?  What  I  hope 
you  are  sensing  Is  that  this  Is  a  vary  different  and  such  more  difficult 
task.  In  our  experiments.  It  takas  about  60  trials  to  become  accurate  at 
performing  the  task.  If  all  you're  doing  Is  filling  In  the  answers,  if  you 
want  to  do  this  task,  and  keep  track  of  where  you  were  In  the  circuit,  and 
not  lose  your  temporary  Information,  It  takes  about  1200  trials  (Cor  I  Ion  & 
Schneider,  1077).  The  normal  clasaroom  rarely  provides  that  much  practice. 
With  years  of  practice,  a  technician  might  get  the  training  trials  needed. 

To  make  to  a  component  skill  good  enough  that  you  can  build  other  tests  on 
top  of  that,  requires  this  extended  training.  You  have  to.  In  my  terms, 
make  that  component  skill  automatic.  Than  you  can  Integrate  It  and  use  it 
In  a  higher  level  of  skill.  Just  being  able  to  perform  the  task  at  the  low 
level  Is  of  marginal  utility. 

An  Important  point  to  note  Is  that  human  working  memory,  where  you 
actually  manipulate  the  material.  Is  somewhere  between  three  and  four 
elements  —  a  vary  small  number  of  elements.  If  you  have  to  recall  rules 
like  the  operation  of  NANO  gates  while  you're  trying  to  engage  In  some 


problem-solving  tesk,  you  have  eliminated  half  your  memory  capacity.  In  a 
rail -world  setting,  that  ollailnatlon  of  memory  capacity  can  have  a  very  big 
Impact  on  perforaiance.  oven  in  a  situation  that  la  not  apeed-st reseed.  In 
our  electronic  troubleehoot Ing  claeaes  the  gates  aren't  that  hard  to  learn 
Individually.  We  eee  a  student  going  through  a  circuit  very  nicely  until 
they  hit  a  poorly  practiced  HAND  gate,  and  all  of  a  sudden  you  can  eee  their 
memory  deteriorate.  They  start  randomly  poking  around,  aiaklng  measurements 
In  the  syetem. 

To  build  up  a. skill  usable  In  the  world  we  must  also  worry  about 
workload.  Being  accurate  Is  not  enough.  Being  accurate  and  performing 
under  a  heavy  workload  Is  a  critical  character  let Ic  In  many  cases.  Let  me 
Illustrate  high  workload  with  a  detection  task  where  subjects  are  asked  to 
search  for  words  from  a  semantic  category.  This  Is  the  task:  If  you  see  a 
word  from  the  categories  awnmala,  birds,  and  fruits,  tap  with  your  left 
hand,  or  the  digit  6,  2,  or  0.  with  your  right  hand.  A  word  and  four  digits 
are  preeented  every  .8  seconds.  None  of  us  can  do  this  Initially.  Subjects 
react  to  the  task  saying  It  is  Impossibls.  However,  after  eight  hours  of 
training,  you  can  perform  digit  and  semantic  search  task  as  well  together  as 
individually  (see  figure  2). 

Let's  again  compare  the  controlled  and  automatic  modes,  using  this 
task.  If  there  Is  a  consistent  response  mapping,  say,  you're  always  pushing 
a  button  to  anlaials,  that  decrement  In  performance  for  the  word  search  drops 
off  to  the  point  of  nearly  zero  with  the  digit  task  added  (figure  2  right 
side).  Therefore,  you  can  add  another  task  on  top  of  the  Initial  task. 
However,  with  a  varied  sapping  the  decrement  In  performance  of  doing  the 
word  search  task,  given  you  also  do  the  digits,  Is  roughly  60  percent 
(figure  2  left  side).  You  can  practice  this  for  as  long  as  you  want. 
Performance  will  not  Improve.  In  fact.  It  gets  slightly  worse. 

How  long  does  the  consistent  mapping  task  take  to  learn?  Oh,  about 
2000  trials,  eight  houre  of  training  In  terms  of  doing  the  task,  so  It’s  not 
cheap.  There's  no  free  lunch  In  this  business;  but  you  can  substantially 
Improvs  performance.  If  you  want  somebody  to  operate  reliably  In  a 
hlgh-workload  situation,  that  extended  amount  of  training  Is  going  to  be 
necessary. 

He  have  found  that  the  first  time  we  Introduced  almost  any  new  task, 
the  task  Interferes  with  any  other  novel  task.  But  after  a  little  practice, 
(e.g.,  60  trials),  subjects  start  doing  reasonably  on  the  other  tasks. 
Processing  resources  becosie  available,  that  Is,  some  other  tasks  can  be  done 
at  the  same  time  with  little  deficit.  Be  are  trying  to  make  lawful 
relationships  between  practice  and  resource  load.  Be  hope  to  provide  the 
training  designer  algorithms  to  specify  when  a  task  Is  sufficiently 
automatic  to  be  performed  reliably  in  a  high  workload  environment. 

The  conclusions  are.  Initial  acquisition  of  almost  any  task  is 
extremely  resource-demanding,  but  with  continued  training,  the  resource 
demand  reduces,  allowing  other  tasks  to  be  combined  without  deficit. 

An  Important  theme  hers  Is  that  It's  not  appropriate  to  ask,  "Is  part 
training  better  than  whole  training?”  That's  the  wrong  way  to  ask  the 
question.  One  potential  question  Is  whether  there  Is  some  character  let Ic  of 
the  resources  needed  to  perform  the  task,  that  can  also  decide  when  to 
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Figure  2.  Reduction  in  dual  task  decrement  across  practice  sessions  as  a  function 
of  dual  task  controlled  processing  versus  dual  task  automatic  processing.  (Adapted 
from  Schneider  &  Fisk,  1984) 
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promote  from  part  training  to  whole  training?  If  you  never  give  the  person 

a  way  to  organize  the  task,  they  can  train  for  a  very  long  time,  and  not  get 

any  better  at  It. 

Some  military  training  programs  like  to  put  trainees  In  a  high-workload 
situation,  and  always  keep  them  at  maximum  workload.  We  studied  a  version 
of  this  using  a  dual  task.  We  gave  one  group  of  people  a  ship  control  task 
and  a  one-digit  search  task.  The  digit  task  was  given  priority  and  which 
they  were  never  to  miss  a  digit.  That  was  a  modest  workload,  that  you 
couldn't  trade  off..  Another  group  doing  the  same  ship  control  task  had  a 
three-digit  search  task,  and  we  said,  "We  don't  care  about  the  digit  task. 

Do  as  much  as  you  can  on  It.” 

The  subjects  under  the  aedest  workload,  although  they  tried  hard,  never 

got  any  better  at  the  task.  Doing  the  dual  tasks  was  taxing  and  they 

couldn't  learn  the  Individual  tasks.  In  contrast,  the  subjects  who  were 
allowed  to  trade  off  learned  both  tasks.  The  first  let  the  digit  task  go 
(l.e.,  missed  the  digits)  and  learned  to  control  the  ship.  Then  they 
attended  to  the  digit  task  and  were  able  to  perform  both  better  than  the 
subjects  who  had  to  maintain  the  digit  task. 

The  implication  here  Is,  If  you  put  In  all  the  parts  of  the  task  too 
early,  you  can  lose  a  student.  That  is,  you  give  them  two  tasks  to  do  at 
once,  while  they  really  only  have  the  capacity  to  do  one,  and  they  won't  let 
one  of  them  go.  They  can  continue  to  practice,  and  will  show  little  benefit 
or  Improvement.  I  think  this  might  be  particularly  Important  when  you're 
dealing  with  people  who  have  differential  abilities.  For  example,  If  you 
throw  somebody  who  doesn't  know  English  well  (still  dealing  with 
understanding  Individual  words)  Into  a  class,  and  expect  them  to  operate  at 
the  same  pace  as  native  speakers,  they  may  work  Just  as  hard,  but  can't  show 
a  normal  acquisition.  They  look  less  Intelligent,  although  that  Isn't  a 
difference. 

Let  me  turn  now  to  processing  reliability.  We  want  people  who  can 
perform  well,  perhaps  when  they  haven't  slept  for  10  hours,  or  20  hours,  or 
they  haven't  recently  trained  In  the  task  (Fisk  t  Schneider,  1082). 

We  looked  at  the  distinction  between  controlled  and  automatic  processes 
as  a  function  of  alcohol  level.  Subjects  were  given  sufficient  alcohol  to 
be  legally  drunk  (.IX  blood  alchohol  level).  In  the  varied  task,  we  had  a 
28  percent  decrement  In  performance;  In  the  automatic  task,  we  had  no 
decrement.  As  a  matter  of  fact,  sometimes  we  got  a  little  Improvement. 

The  comment  of,  "You've  drunk  a  lot,  and  you  can  drive  home 
successfully"  —  I  won't  say  safely,  but  successfully  —  probably  has  a  fair 
amount  of  truth  to  It.  If  nothing  unusual  happens.  If  you're  consistent, 
and  you  use  the  normal  path,  you  can  be  quite  reliable.  In  terms  of  the 
processing.  If  anything  unusual  happens,  then  you're  in  deep  trouble. 

Similar  results  have  have  been  obtained  for  heat  stress  and  sleep 
deprivation  (see  Hancock,  1084).  Controlled  processing  Is  quite  susceptible 
to  these  stressors,  and  automatic  processing  Is  much  less  susceptible. 

There  are  some  problems  associated  with  automatic  processing.  One  Is 
negative  transfer.  For  example.  If  the  location  of  the  control  that  lowers 
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the  landing  gear  Is  moved  after  years  of  training,  pilots  may  experience 
negative  transfer.  In  a  stress  situation,  they  drop  back  to  the  old 
procedure. 

In  the  laboratory  context,  we  examined  negative  transfer  with  a  search 
task.  We  said,  "Always  push  the  button  when  you  see  A,  B,  C,  or  D , ■  and 
then  we  switched  them  over  and  said,  "Now,  always  push  the  button  when  you 
see  X,  Y,  or  Z,  but  Ignore  A,  B,  C,  and  D."  In  the  automatic  mode,  It  took 
three  times  longer  to  break  the  skill  than  It  took  to  develop  It  (Shlffrin  & 
Schneider,  1977).  I  might  add.  It  took  thousands  of  trials  to  develop  It. 

In  a  controlled  task.  It  turns  out  It  took  roughly  the  same  amount  of  time 
both  to  develop  It  and  to  break  It.  This  is  difficult  to  evaluate  because 
of  errors  so  close  to  zero.  There's  a  risk  with  the  automatic  mode  that 
responses  may  occur  even  when  they're  not  Intended.  Figure  3  Illustrates 
the  difference  In  Initial  and  reversal  training  for  controlled  and  automatic 
processing.  Controlled  processing  required  very  few  trials  for  acquisition 
and  reversal.  In  contrast,  automatic  processing  required  many  more  trials 
for  acquisition.  However,  store  Importantly,  reversal  training  required 
nearly  three  times  as  many  trials  as  Initial  acquisition. 

From  these  results,  I  conclude  that  training  programs  must  be  designed 
carefully  focusing  on  high  performance  skills,  particularly,  those  that 
occur  In  h Igh-work load  situations.  Suppose  the  task  Is  one  where  the 
demands  could  change  and  have  the  potential  of  negative  transfer.  In  such 
environments,  one  should  redesign  the  task  If  possible.  Note  that  negative 
transfer  effects  are  most  severe  In  very  high  workload  situations.  They  are 
not  that  severe  In  low  workload  situations.  To  avoid  negative  transfer  in 
situations  that  demand  change,  an  upper-level  goal  state  or  context  cue 
could  be  built  In.  In  one  context,  do  this,  and  In  another,  do  that. 

Another  problem  characteristic  sssoclated  with  automatic  processing  Is 
loss  of  control.  Apparently  subway  motormen  hardly  ever  make  an  error 
dealing  with  the  signals,  such  as  stopping  at  the  station  (assuming  they  are 
awake).  However,  If  they  are  asked  to  "Skip  the  next  station,”  they 
Invariably  stop  (Haber  t  Haber,  1985).  If  you've  tried  to  stop  at  a  store 
while  driving  home  and  gotten  distracted,  you  know  this  characteristic  of 
automatic  processing.  You've  lost  control.  That  loss  of  control  is 
necessary  to  enable  low-resource  performance.  So  It  Is  good.  But  there  are 
certain  situations  where  It  can  be  counterproduct i ve. 

The  third  characteristic  of  problem  areas  for  automatic  processing  is 
Mmory  modification.  How  aiuch  do  you  learn  when  you  execute  an  automatic 
process,  versus  a  control  process? 

We  have  looked  at  the  frequency  with  which  words  can  be  detected  with 
automatic  or  control  processing  (Fisk  &  Schneider,  1984).  We  had  subjects 
perform  a  semantic  search  task  (l.e.,  look  for  vehicle  words)  and  present 
words  one  to  twenty  times.  If  subjects  performed  a  controlled  search  task, 
subjects  reported  that  a  word  presented  twenty  times  had  an  occurrence  of 
eighteen  times.  In  contrast,  subjects  doing  automatic  search  had  a  reported 
estimate  of  zero  occurrences  of  a  word  presented  twenty  times.  Note,  the 
subjects  were  semantically  proceaslng  the  words  sufficiently  to  detect  novel 
vehicle  words. 
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These  results  heve  iaportsnt  Imp  t lest  Ions  for  the  trslnlng  context,  if 
you  si  low  people  to  learn  by  rote  rehearsal  of  the  material,  they  may  need 
very  little  memory  modification  In  terms  of  the  process.  The  ability  of  the 
person  to  actually  remember  what  they  did,  versus  what  they  should  have 
done,  may  be  greatly  reduced. 

To  Just  summarize  these  effects,  I've  talked  about  the  large 
qualitative  differences  (See  Fisk,  Schneider  fc  Logan,  1987  for  details).  In 
varied  aupplng,  you  see  little  Improvement  with  practice.  In  consistent 
mapping,  you  see  very  substantial  Improvement.  So  that  the  place  to  put  a 
substantial  amount  of  your  training  tla*,  Is  In  these  consistent  components 
(See  Schneider,  1985  for  review). 

I've  tslked  about  the  processing  speed-up.  We’ve  had  a  hundredfold 
Increase  In  processing  speed.  I've  talked  about  reduced  workload,  and  we 
can  suike  a  task  so  that  It  has  almost  no  load.  This  Is  Important  both  In 
the  contexts  of  high-workload  tasks,  or  tasks  In  which  you're  going  to  build 
another  skill  on  top  of  It,  like  electronic  troubleshooting.  I’ve  talked 
about  processing  reliability.  We  want  this  task  to  go  on,  even  If  the 
person  Isn't  in  the  best  of  shape.  I've  talked  about  the  negative  effects, 
that  we  can  have  a  greater  problem  of  Interference  effects.  We  can  have 
some  loss  of  control,  and  less  memory  modification,  for  these  automatic 
processes. 

I  have  a  aiodel  which  assumes  that  processing  goes  through  four  phases 
(Schneider,  1985).  I  think  most  classroom  training  gets  to  the  second 
stage,  still  basically  In  the  control  processing  mode.  In  many  tasks, 
reaching  the  third  stage,  where  you  have  to  at  least  generally  attend  to  the 
task,  can  be  Important.  To  do  that  typically  requires  a  substantial  amount 
of  training.  The  third  stage  begins  after  about  50  trials  of  doing  the 
task.  For  example,  for  many  of  the  component  tasks  performed  by  air  traffic 
controllers  or  sir  intercept  controllers,  they  hsven’t  had  50  trials  of 
doing  that  task  even  after  a  six-week  training  program. 

There  Is  a  crisis  and  a  challenge  In  skill  training.  It's  a  challenge, 
because  we  know  that  with  the  right  kind  of  practice  we  produce  dramatic 
performance  Improvements.  We  can  do  It  In  the  laboratory  setting,  and  in 
certain  applied  settings. 

It's  a  challenge  because  It  takes  a  lot  of  time.  After  ten  years  of 
experience  on  the  Job,  you  can  see  these  performance  changes.  Compare  the 
expert  and  the  novice  air  traffic  controller.  The  expert  sits  there  quietly 
making  commands  without  looking  overloaded.  The  novice  Is  sweating 
profusely,  can't  figure  out  what  Is  going  to  happen  next,  and  Is  overloaded. 
How  do  we  compress  experlenes?  The  military  has  this  problem,  and  it 
doesn't  have  the  luxury  of  waiting  ten  years  to  grow  a  new  controller. 

In  aklll  acquisition,  from  my  perspective,  automatic  component  skills 
must  be  developed  for  those  tasks  that  keep  reappearing.  A  critical  aspect 
of  training  Is  building  up  those  components,  and  developing  the  time-sharing 
skills  that  allocate  control  processing  to  the  Inconsistent  or  poorly 
learned  task.  For  example,  In  the  case  of  electronic  troubleshooting, 
keeping  track  of  the  voltage  levels  throughout  the  circuit.  People  need 
some  experience  In  that,  but  It  doesn't  help  to  give  them  a  lot  of 
exper  fence. 


One  of  the  processes  that  we  are  using  to  facilitate  automatic  control 
is  this  time-compressed  training.  We  provide  the  learner  with  many 
consistent  trials  performing  the  task.  In  the  case  of  air  traffic  control, 
the  critical  aspect  for  air  Intercept  control  Is  bringing  two  airplanes 
together.  The  critical  task  Is  projecting  the  trajectories  of  those 
airplanes  and  getting  them  to  the  right  point.  If  you  Intercept  In  real 
tlaie  In  the  simulator,  a  single  Intercept  takes  about  20  minutes.  In  this 
case,  we  can  compress  time  by  a  factor  of  100:1.  Experience  with  2,000 
maneuvers  In  each  of  the  particular  areas  of  air  Intercept  control  could 
take  five  or  ten  years  on  the  Job.  We  can  provide  2,000  trials  on  each  of 
the  critical  components  In  compressed  time  In  about  a  week.  We  practice  the 
components  and  then  deal  with  the  total  task.  This  does  not  suggest  that 
normal  procedure  Is  eliminated.  Rather,  Initial  training  Is  performed  at  a 
time  compression  that  is  optimal  for  learning,  not  necessarily  the  time 
scale  of  the  real  world. 

It  Is  difficult  to  do  2,000  trials  of  anything,  and  keep  going. 
Therefore  motivation  Is  a  very  Important  part  of  training.  In  my  own  case, 
the  most  cost-effective  piece  of  equipment  In  my  lab  la  a  $15  noise 
synthesis  chip  that  makes  Interesting  sound  effects.  It  keeps  subjects 
going  after  thousands  of  trials.  In  the  electronic  troubleshooting  task,  we 
use  competition  to  keep  students  working  for  weeks. 

I'll  Just  briefly  present  some  of  the  guidelines  for  designing  training 
programs  (See  Schneider  1085,  Reg  I  an  t  Schneider,  1087).  The  first  one  Is 
"Present  information  to  promote  consistent  processing."  There  are  a  number 
ways  of  doing  that.  One  Is  that  you  can  use  a  verbal  rule.  In  the  case  of 
the  electronic  gates.  It's  on  If  any  Input  Is  on.  You  can  use  an  analogy. 
You  can  provide  examples.  You  can  provide  a  table.  The  function  of  all  of 
this  Is  to  produce  a  consistent  execution  during  the  first  phase  of 
training.  If  the  student  doesn't  see  the  consistency  of  how  to  deal  with 
that  task,  you  are  not  going  to  see  much  of  an  Improvement  In  practice.  In 
many  complex  tasks,  for  example  In  air  traffic  control.  It  Is  not  sasy  to 
see  those  consistencies.  You  have  to  design  the  tasks  so  that  the 
consistencies  are  apparent. 

Next,  "Introduce  novel  components  singly,  and  do  not  overload 
processing  during  Initial  acquisition."  Now,  remember  that  If  Initial 
resources  do  not  exist,  extended  practice  will  lead  to  little  Improvement  in 
performance.  With  the  available  resources,  you  see  a  reasonable  performance 
Improvement  function. 

The  third  point  Is,  "Design  the  task  to  allow  many  trials  of  the 
critical  skill."  In  the  electronic  troubleshooting,  for  example,  we  give 
them  a  lot  of  trials  In  dealing  with  gates.  In  terms  of  a  simple  prediction. 
The  goal  Is  to  decrease  response  time  as  a  function  of  training.  More 
Importantly,  many  real  world  tasks  must  also  be  accomplished  under  high 
workload.  In  the  case  of  electronic  troubleshooting,  that  workload  Is 
Imposed  by  keeping  track  of  other  elements  of  the  task. 

Four,  "Maintain  motivation  throughout  training."  With  training  using 
hundreds  of  trials,  motivation  Is  very  Importsnt.  With  subjects  In  the  lab 


we  find  that  If  aw  don’t  worry  about  the  motivation,  they  aril  I  show  a  nice 
Improvement  function  for  a  while,  but  after  about  the  fourth  or  fifth  hour, 
learning  levela  off.  Deal  training  program*  generally  all  go  much  beyond 
four  or  five  hour*. 

Last,  "Aa  automat Iclty  develops,  are  utilize  the  available  resources  to 
Integrate  and  enhance  component  tasks.”  So  that,  as  the  resources  become 
available,  training  can  use  those  available  resources.  In  the  electronic 
troubleshoot Ing,  training  moves  from  gates  alone  to  training  In  the  circuit. 
In  eome  aense,  training  time  Is  arasted  If  you  spend  too  much  time  with 
components. 

Human  performance  changes  drastically  with  practice.  I  believe 
behavior  le  the  Interaction  between  two  quantitatively  different  forms  of 
processing,  referred  to  as  automatic  and  controlled.  This  distinction  has 
Implications  for  training.  In  particular  It  suggests  that  we  focus  practice 
on  the  consistent  components  of  the  task.  We  design  the  training 
environment  to  allow  many  training  trials,  and  we  promote  to  higher  levels 
of  skill  training  on  the  basis  of  workload.  In  terms  of  processing. 
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I'm  going  to  talk  about  a  very  different  transfer  problem,  the  climate 
within  an  organization  which  may  Inhibit  or  facilitate  transfer  of  training. 
But,  first,  a  1 1 tt le  background. 

Since  going  to  my  new  life  as  a  department  head  at  North  Carolina 
State,  l  have  had  a  number  of  students  working  on  Interpersonal  skills  for 
managers  —  teaching  how  to  motivate,  how  to  set  goals,  how  to  discipline, 
ail  these  different  kinds  of  things. 

We  use  a  behavior-modeling  training  program,  based  rather  loosely  on 
Bandura's  social  learning  theory,  concerned  with  teaching  Interpersonal 
skills.  For  a  variety  of  reasons,  this  kind  of  training  Is  less  likely  to 
transfer  than  the  typical  kind  of  skill  training.  There  may  be  factors 
within  the  organization  which  prevent  transfer:  expectations  of 
subordinates  as  to  how  the  trainee  should  behave,  production  demands,  etc. 

I  want  to  focus  on  these  kinds  of  problems,  Instead  of  the  traditional 
transfer  problems. 

In  Interpersonal  skills  training,  as  In  all  training,  It's  very  clear 
that  you  need  to  do  needs  assessments.  Unfortunately,  In  training  In  the 
life  Insurance  Industry  (where  I  worked  for  21  years).  In  almost  any 
Industry,  and  most  likely  In  the  military,  there’s  an  awful  lot  of  training 
that  gets  done  without  any  needs  assessment  whatsoever.  This,  of  course, 
results  In  faddlsm.  Quite  frankly,  I  think  the  Army's  move  Into 
organizational  effectiveness  was  In  part  motivated  by  some  faddlsm  as 
opposed  to  a  basic  and  thorough  needs  assessment.  As  Bob  Mager  has  pointed 
out  so  well,  many  performance  deficiencies  are  not  training  problems.  They 
are  selection  problems,  or  motivational  problems,  or  some  other  kind  of 
problems.  We  do  any  awful  lot  of  training  when  we  should  be  doing  something 
else. 


As  you  know,  needs  assessment  should  tell  us  what  the  goals  and 
objectives  of  our  training  might  be,  the  criteria  to  be  used  In  the 
evaluation,  and  the  content  and  the  methods  of  training.  Bill  McGehee  and  i 
have  developed  a  basic  trilogy  for  needs  assessment:  organizational 
analysis,  task  analysis,  and  person  analysis.  Because  task  analysis  and 
person  analysis  are  so  familiar,  I  want  to  spend  more  time  talking  about 
organization  analysis. 
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l7T  organization  analysis,  the  organization's  objectives,  resources,  and 
the  climate  for  training  or  transfer  are  carefully  analyzed.  The 
organizational  creed,  or  goal  statement  Is  examined.  Organizational 
resources  in  terms  of  humans  and  equipment  are  examined.  The  acceptance  of 
training  within  the  organization  Is  evaluated,  and  so  forth.  Unfortunately, 
getting  all  this  Information  and  putting  It  together  Is  an  art,  not  a 
science.  Thus,  our  training  may  fail  short  and  fall  to  transfer. 

When  we  find  out  there's  a  need,  we  train,  we  test,  we  find  out  what 
people  know  what  are,  taught  them  In  the  training.  We  ask  them  to  behave  In 
the  way  we  taught  them  to  behave  In  the  training.  But  often,  when  we  look 
at  them  on  the  Job,  nothing  happened.  We  wonder  what's  wrong.  What 
prevented  transfer?  SometiMS  we  find  the  behavior  did  transfer,  but  It  had 
no  effect.  What  happened?  Were  there  organizational  factors  that  prevented 
transfer? 

Let  sie  give  you  an  example.  In  20  years.  30,000  foreign  engineers  were 
trained  In  the  United  States,  with  the  objective  to  train  them  here  and  have 
them  return  to  their  countries  of  origin.  Well,  those  engineers  are  still 
here,  competing  with  American  engineers.  An  appropriate  organization 
analysis  could  have  shown  the  foreign  engineers  were  being  trained  to  work 
In  an  environment  that  doesn't  exist  In  their  own  country. 

Another  example:  Police  training  typically  emphasizes  the  use  of 
firearms,  filling  out  reports,  and  legal  requirements.  The  task  analysis 
shows  this,  but  an  organization  analysis  would  point  out  organizational 
expectations  about  Interpersonal  relationships  with  the  public,  positive 
Image,  et  cetera.  If  you  don't  Include  these  In  your  training,  the  training 
will  fall. 

Or.  look  at  the  training  of  the  hard-core  unemployed  which  occurred 
many  years  ago.  They  were  taught  how  to  perform  on  a  Job,  not  about  other 
expectations,  such  as  being  punctual,  working  every  day,  respecting  the 
foreman,  and  so  forth.  If  these  skills  aren't  taught,  the  Job  skills  won't 
have  a  chance  to  transfer. 

Let  me  discuss  one  particular  aspect  of  all  of  this:  the  climate  for 
transfer.  |  will  be  borrowing  heavily  from  Irv  Goldstein  In  what  follows. 

Basically,  we  want  to  learn  about  organizational  facilitators  and 
Inhibitors  of  transfer.  With  such  Information,  training  can  be  modified  to 
Innoculate  employees  against  potential  problems. 

We  know  how  to  go  out  and  get  knowledge  and  skills  and  attitudes 
required  on  the  job.  We're  not  very  good  at  providing  Inputs  to 
Instructional  designers  concerning  the  transfer  climate. 

To  overcome  this  deficiency,  Irv  Goldstein  Is  attempting  to  develop  a 
method  for  aystsmat leal ly  assessing  transfer  climate.  The  same  subject 
matter  experts  used  for  developing  task  analyses  will  be  Interviewed  to  find 
out  about  facilitators  and  Inhibitors  for  transfer. 

He  Is  using  the  Katz  and  Kahn  open-system  model.  Any  organization, 
according  to  that  model.  Is  an  open  system  which  has  five  major  dimensions: 
support,  maintenance,  production,  adaptive,  and  managerial  dimensions.  The 


support  system  Is  Important  In  providing  support  for  the  resources  of  the 
organization.  Maintenance  are  those  various  ways  of  attaching  the  people  to 
work,  keeping  the  system  going.  Production  Is  the  actual  dlaplay  of 
performance.  The  adaptive  system  Is  the  long-term  provision  within  the 
organization  for  adapting  to  change  as  times  and  conditions  change  In  the 
outside  world.  The  managerial,  of  course,  is  the  whole  business  of 
supervising  policy  coordination  and  control. 

For  example,  under  maintenance,  you  might  ask.  Is  training  made 
available  when  you  need  It?  Do  employees  leave  because  developmental 
training  Is  not  available?  Under  production,  do  Job  Incumbents  get  a  chance 
to  practice  learned  behavior  In  the  organization,  or  does  It  only  come  about 
In  the  event  of  an  emergency?  Do  employees  In  our  department  use  safety 
behaviors  taught  to  them  In  training  when  It  Interferes  with  a  production 
standard? 

If  we  can  accurately  asseas  the  climate,  what  do  we  do  from  that  point? 
If  we  find  we  have  a  negative  climate  for  transfer,  what  kind  of  strategies 
or  options  do  we  have  at  that  point? 

There  are  a  number  of  very  definite  approaches  to  take  with  management. 
Trained  behaviors  must  be  reinforced  on  the  Job.  A  trainee  who  comes  back 
Into  an  unappreciative  environment  for  ski  I  la  trained  will  be  Ineffective. 
Strategies  siust  be  taught  which  will  enable  the  trainee  to  transfer  skills 
despite  that  environment.  Another  problem  occurs  when  the  skills  taught  are 
not  used  for  several  sionths.  What  can  we  teach  the  trainee  to  keep  the 
skills  "al Ive?* 

I  think.  If  you  have  an  accurate  assessment  of  the  transfer  climate, 
you  may  be  able  to.  In  essence.  Innoculate  the  trainee  —  or  provide  support 
systems  for  the  trainees  so  that  even  If  the  organization  Itself  doesn't 
Change,  the  trainees  can  go  there  with  the  appropriate  mechanisms  to  make 
sure  that  the  behavior  does  transfer. 

Irv'a  work,  which  I  plan  to  try  out.  Is  still  In  the  early  stages  of 
development,  but  I  believe  It  Is  the  first  systematic  breakthrough  In  an 
Important  area.  I  think  It  will  help  give  us  better  understanding  and  a 
means  of  anticipating  transfer  problems  more  systematically.  That  will 
provide  us  with  the  means  to  avoid  or  overcome  Inhibitors,  and  to  capitalize 
on  fact  I Itators. 


v.v.v 
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I  would  like  to  consent  on  five  specific  areas  within  the  general 
context  of  transfer  of  training  research.  Some  of  these  areas  are  very 
broad,  others  are  wore  specific. 

1.  Let  ste  begin  by  saying  that.  In  my  opinion,  the  term  'training"  is 
so  broad  as  to  severely  confuse  many  of  the  discussions  that  are  devoted  to 
It.  Training  In  art  appreciation  Is  very  different  from  training  In 
electronic  troubleshooting.  Training  in  basketball  foul-shooting  Is  very 
different  from  training  In  cognitive  psychology.  Training  In  woodcarving  is 
very  different  from  training  In  18th  century  French  literature.  In  my 
opinion,  preferred  approaches  to  training,  and  therefore  to  transfer,  are 
Inexorably  confounded  with  content.  This  Is,  of  course,  not  new.  A  lot  of 
people  have  made  this  point  previously.  He  have  discussed  It  here  at  some 
length. 

A  major  problem  In  the  area  of  training  and  education  that  has  been 
discussed  literally  for  centuries.  Involves  the  relative  Importance  of 
pedagogy  versus  content.  In  my  opinion,  this  Issue  Is  far  from  being 
resolved,  and  continues  to  haunt  our  investigations  of  transfer  of  training, 
both  In  the  military  and  In  civilian  environments.  I  personally  feel  that, 
In  general,  we  do  not  adequately  provide  for  mastery  of  the  necessary 
content  material  by  Instructional  developers  prior  to  and  during  the 
development  of  training  programs.  In  many  military  training  environments 
with  which  1  am  familiar,  Instructional  developers  are  not  themselves 
subject  matter  experts.  Further,  In  my  opinion,  they  do  not  adequately 
consult  subject  matter  experts  In  their  development  activity.  I  feel  that 
Instructional  developers  should  be  subject  matter  experts  who  are  trained  in 
pedagogy  or  education  (or  who  bring  Instructional  experts  In  from  the 
outside),  not  vice  versa.  In  my  opinion,  this  Is  a  big  mistake. 

Of  course.  It's  possible  for  a  smart  person  to  learn  either  (or  both) 
of  these  disciplines.  However,  It  could  take  years  for  a  training  expert  to 
master  a  unique  content  domain,  such  as  International  mortgage  banking,  for 
example.  Bo.  where  does  this  leave  those  of  us  who  are  In  the  training 
research  community?  In  my  opinion,  we  need  to  develop  Interactive  design 
taxonomies,  which  combine  such  approaches  as  the  Department  of  Labor's 
occupational  codes,  with  the  types  of  tasks  and  skills  Involved,  and  overlay 
this  with  appropriate  training  strategies  for  each  unique  combination. 

Until  we  start  working  toward  something  like  this,  we  are  going  to  be 
continually  criticized  for  not  having  our  act  together,  and.  In  my  opinion. 


criticized  with  some  Justification.  Research  has  been  conducted  for  decades 
In  this  area,  and  a  lot  of  whee l -spinning  and  reinvent  Ion  Is  going  on.  We 
need  to  codify  this  vast  training  literature  In  some  defensible  way.  so  that 
we  etop  reinventing  the  wheel,  and  can  generalize,  and  so  that  we  don't  have 
to  conduct  unnecessary  transfer  of  training  studies.  In  areas  that  we 
already  know  about. 

2.  Secondly,  we  need  to  arrange  the  situation  so  that  hardware  and 
software.  In  the  training  development  world,  does  not  drive  training,  but 
vice  versa.  This  Is ,  In  my  opinion,  a  big  problem  area.  The  potential 
power  of  the  new  technology  In  training  and  Instruction  (such  as  teaching 
aiachines,  videodiscs,  simulators,  CAI,  so-called  "Intelligent*  CAI, 
artificial  Intelligence  techniques,  etc.)  Is  awesome,  but  we  can't  let  It 
drive  our  Instructional  development  efforts. 

I  recently  ran  Into  what  I  consider  to  be  a  very  cogent  article  on  this 
topic  by  Derek  Bok,  who  le  the  President  of  Harvard  University.  This 
article  was  excerpted  from  Bok's  annual  report  to  the  Harvard  Board  of 
Overseers.  ( I ,  by  the  way,  was  amazed  that  the  President  of  Harvard 
University  would  devote  his  report  to  the  Board  of  Overseers  to  the  topic  of 
transfer  of  training.)  In  that  article,  Bok  cited  a  number  of  observers  of 
the  training  and  educational  technology  scene.  In  what  Is  essentially  a 
"good  news/bad  news*  type  of  format.  Let  me  extract  from  some  of  his 
comments,  for  the  purpose  of  our  discussion  here. 

First,  the  good  news.  According  to  Steven  Muller,  President  of  Johns 
Hopkins  University,  due  primarily  to  the  advent  of  CAI,  "We  are,  whether 
fully  conscious  of  It  or  not,  already  In  an  environment  of  higher  education 
that  represents  the  moat  drastic  change  since  the  time  of  the  University  of 
Paris,  some  eight  or  nine  centuries  ago."  Similarly,  according  to  Ray  Neff, 
Director  of  Computer  Sciences  at  Dartmouth,  "Because  of  the  speed  and 
accuracy  of  the  computer  In  performing  computations  and  processing 
Information,  what  Ph.D.'s  did  25  years  ago  can  be  term  projects  for 
Dartmouth  students  today."  And,  over  twenty  years  ago,  Patrick  Suppes  made 
the  following  statesient:  "One  can  predict  that,  In  a  few  store  years, 
millions  (of  students)  will  havs  access  to  what  Philip  of  Macedon's  son 
Alexander  enjoyed  as  a  royal  prerogative:  The  personal  services  of  a  tutor 
as  we  I l-lnfora»ed  and  as  responsive  as  Aristotle." 

Bok  goes  on  in  his  article,  however,  to  temper  this  euphoria  with  a 
certain  amount  of  skepticism  (l.e..  Now,  the  bad  news).  According  to 
Richard  Clark  at  the  University  of  Southern  California,  "Five  decades  of 
research  suggest  that  there  are  no  learning  benefits  to  be  gained  from 
employing  different  media  In  Instruction,  regardless  of  their  obviously 
attract Ivs  features  or  advertised  superiority.  The  best  current  evidence  Is 
that  media  are  mere  vehicles  that  deliver  Instruction,  and  do  not  Influence 
student  achievement  any  more  than  the  truck  that  delivers  groceries  causes 
changes  In  our  nutrition." 

Bok  himself  states  that,  "Ezperlencs  should  make  us  wary  of  dramatic 
claims  for  the  Impact  of  the  new  technology.  Thomas  Edison  was  clearly 
wrong  when  he  declared  that  the  telegraph  would  revolutionize  education. 
Radio,  also,  did  not  make  a  lasting  Impact  on  the  public  schools,  even 
though  foundations  gave  generous  subsldlss  to  bring  programs  Into  the 
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classroom.  Television  met  s  similar  fate  In  spite  of  glowing  predictions 
heralding  Its  powers  to  Improve  learning.” 


In  each  Instance,  according  to  Bok,  "Technology  failed  to  live  up  to 
Its  early  promise  for  three  reasons:  Resistance  by  teachers,  high  cost,  and 
(most  Importantly)  the  absence  of  demonstrable  gain  In  student  achievement. 
There  Is  as  yet  no  clear  evidence  that  computers  and  videodiscs  will  meet  a 
happier  fate." 


According  to  Bok,  "The  educational  benefits  of  technology  remain  In 
dispute.  There  Is  still  little  proof  that  these  devices  yield  lasting 
Improvements  In  learning.  Many  studies  purport  to  find  such  gains.  But, 
most  of  them  can  be  explained  on  the  grounds  that  students  using  computers 
were  temporarily  activated,  by  the  sheer  novelty  of  the  machines,  or  that 
SK>re  effort  and  better  teaching  went  Into  the  computerized  courses  than  were 
devoted  to  the  conventional  classes  with  which  they  were  compared.  Thus, 
learning  Isiprovements  that  early  Invest Igators  reported  shrank  to  virtually 
nothing,  when  the  same  teacher  taught  both  the  experimental  and  conventional 
classes  with  comparable  amounts  of  preparation.  Similarly,  the  gains 
achieved  In  computer  experiments  lasting  less  than  four  weeks  dropped  by 
more  than  two-thirds,  when  the  experiments  continued  beyond  eight  weeks,  and 
the  novelty  of  the  new  technology  wore  off.  Undaunted  by  such  obstacles, 
educators  and  high-tech  companies  spent  huge  amounts  of  money  to  prove  the 
skeptics  wrong.  Control  Data  Corporation  reputedly  Invested  almost  a 
billion  dollars  In  the  computerized  college  curriculum,  PLATO." 


What  does  all  this  swan?  In  my  opinion,  this  means  that,  at  some  level 
of  detail,  we've  got  to  guard  against  the  tall,  essentially,  wagging  the 
dog.  I  personally  have  some  data  to  support  this  contention.  In  a  recent 
series  of  studies  comparing  Interactive,  computer  based  systems  of 
Instruction,  with  two  such  less  expensive  methods  of  Instructional  delivery 
In  electromechanical  troubleshoot Ing,  no  differences  were  found  In  mastery 
or  transfer  across  Instructional  delivery  methods  when  the  time  spent  In 
training  was  controlled,  and  the  material  was  Identical.  This  occurred  for 
two  tasks,  of  varying  complexity,  both  immediately  following  training  and 
after  a  one-week  retention  period. 


The  point  here  Is  that.  In  my  opinion,  our  research  should  deemphasize 
hardware,  and  should  Instead,  emphasize  the  specific  training  requirements 
for  learning  and  optimal  Instructional  strategies. 


3.  Third,  let  me  turn  to  a  topic  which  I  believe  le  dramatically 
underemphas I  zed  In  our  research  on  training:  the  use  of  concrete  goal 
setting.  I  realize  that  there  Is  widespread  difference  of  opinion  as  to 
what  transfer  Is.  Is  It  the  difference  between  performance  of  trained 
versus  untrained  persons?  Is  It  the  difference  between  obtained  transfer 
and  the  maximum  posalble  transfer?  Is  It  a  measure  of  learning  rate7 
Regardless  of  these  definitions,  all  have  In  common  the  perspective  of  how 
later  performance  le  affected  by  current  training.  One  common  way  of 
enhancing  downatream  performance  which  has  been  widely  researched  In  the 
Industrial  (but  not  the  transfer  of  training)  world.  Is  via  the  application 
of  concrete  goal  setting.  There's  a  body  of  data  that  testifies  to  the 
superior  performance  effects  of  concrete  versus  abstract  goal-setting. 
However,  there's  very  little  work  out  there,  (Insofar  as  I  am  familiar)  on 
this  topic  In  the  domain  of  transfer  of  training. 
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A  singular  exception  is  a  program  that  aras  developed  at  the  Office  of 
Personnel  Management.  It  la  called  PAPA  —  Participant  Action  Plan 
Approach.  PAPA  Is  a  technique  which  requires  that  training  participants 
commit  In  writing  to  concrete  performance  goals,  which  they  themselves 
Initiate,  and  which  ars  based  upon  the  training  they  receive.  The  extent  to 
which  these  goals  are  met  In  the  subsequent  performance  environment  Is  the 
training  evaluation  criterion.  I  want  to  awntlon  this  area  as  one  that  I 
think  may  be  a  potential  gold  mine  for  facilitating  transfer  of  training. 

4.  The  fourth  topic  I  would  like  to  comment  on  Involves  the 
development  and  the  use  of  a»dels,  whose  purpose  Is  to  predict  potential 
transfer  of  training  without  actually  conducting  transfer  experiments.  To 
my  mind,  this  Is  a  major  area  of  endeavor  which  could  produce  substantial 
results  If  such  a»dels  were  thoroughly  developed.  Implemented,  and 
validated.  Models  of  this  sort  have  been  around  for  a  long  time  —  Osgood’s 
transfer  surface  Is  an  example.  However,  they  have  not  been  Implemented  in 
practical  ways  In  an  applied  environment. 

Of  note  are  the  recent  API  and  Naval  Training  Systems  Center  efforts  on 
this  topic.  These  studies  have  Involved  a  series  of  models  formerly  known 
as  TRAINVICE,  and  siore  recently  known  as  DEFT,  (e.g..  Device  Effectiveness 
Forecasting  Technique).  I  have  passed  out  a  report  in  this  area  to 
participants  at  this  steetlng.  Basically,  there  were  several  such  models 
developed  by  a  variety  of  developers.  The  report  that  I  passed  out  Is  a 
commentary  on  four  of  these  andels. 

Presently,  the  research  and  development  needs  of  these  models  Involve 
four  major  problem  areas  as  follows:  (1)  the  theoretical  constructs  of  the 
models,  (2)  their  mathematical  formulat ions,  (3)  measurement  Issues 
Involving  the  validity,  reliability,  and  precision  of  the  models,  and  (4) 
their  convenience  of  application  and  acceptabl 1 1 Ity  to  the  user. 

Regarding  the  theoretical  construct  of  the  aedels,  the  four  TRAINVICE 
models  reflect  the  following  basic  assumptions:  they  attempt  to  find  out 
what  people  know  now,  to  determine  what  they  need  to  know,  to  compute  the 
delta  between  these  two  positions,  and  to  overcome  that  delta  with  training 
techniques  that  deliver  training  content  designed  to  assure  learning. 
Further,  the  models  attempt  to  assess  the  extent  to  which  the  similarity 
between  the  device  and  the  parent  equipment  Is  sufficient  to  permit  adequate 
transfer  to  occur. 

Regarding  the  mathemat leal  formulat Ion,  all  TRAINVICE  a»dels  provide  a 
mathematical  formula  Intended  to  forecast  transfer.  There  Is,  however,  no 
discussion  in  ths  models  to  the  possibility  of  negative  transfer.  Further, 
the  models  assume  that  physical  and  functional  almtiarlty  between  the  device 
and  the  operational  equipment  are  squally  Important  In  facilitating 
transfer. 

It  eeems  to  me  that  there  ars  additional  areas  that  need  to  be 
Investigated.  The  TRAINVICE  models  havs  never  been  validated.  A  variety  of 
studies  nesd  to  be  conducted  to  test  thsse  mode  I s  in  applied  settings. 
Further,  such  ateasurement  models  must  be  convenient  to  implement.  The 
TRAINVICE  models  as  thsy  currently  exist,  are  extremely  Inconvenient  to 
Implement.  This  Is  another  area  that  requires  major  attsntlon. 


'CTW1 


-*  >  .■  j-  -■*  '  •  ' 


Such  approaches  as  the  TRAINVICE  models  are  In  their  Infancy.  There's 
a  lot  of  potential  aierlt  here.  The  recently  revised  DEFT  models  developed 
by  the  Army  Research  Institute  are  In  fact,  a  step  In  this  direction. 

5.  The  fifth  and  final  area  that  I'd  like  to  comment  upon.  Involves 
the  topics  of  technical  documentation  and  job  aids. 


Over  20  years  ago.  Wolf  and  Berry  described  a  Job  aid  as  something  that 
guides  an  Individual  In  the  performance  of  a  Job,  so  as  to  enable  him  to  do 
something  which  he  had  not  previously  been  able  to  do,  without  requiring  him 
to  undergo  complete  training  for  each  task.  Others  have  defined  Job  aids  as 
documents  or  devices  that  store  Information  which  Is  required  to  perform  a 
particular  operation  or  set  of  operations,  and  which  makes  the  Information 
aval  table  on  the  Job. 

Job  aids  differ  In  known  ways  from  other  system  elements,  like  tools 
and  training.  Job  elds  differ  from  tools  in  that  toots  generally  do  not 
store  Information  and  make  it  available  on  the  Job.  Job  aids  differ  from 
training  in  that  training  Is  designed  to  encourage  the  learning  of  a 
particular  skill,  whereas  Job  aids  are  designed  to  assist  In  the  performance 
of  that  skill  in  the  work  environment. 

I'll  mention  briefly  a  couple  of  studies  In  the  Job  aids  realm,  done  by 
Elliott  and  Joyce  In  the  '60s  and  '70's  that  provide  a  fair  amount  of  data 
to  testify  to  the  utility  of  Job  aids,  particularly  in  complex  environments. 

One  study  Involved  a  job  aid  that  was  designed  for  use  by  low-aptitude, 
novice  technicians  on  a  radar  system.  In  that  study,  the  technicians  using 
the  Job  aid  were  capable  of  reducing  the  time  required  to  Isolate  and 
correct  malfunctions  by  as  much  as  SO  percent,  over  the  time  required  by 
highly  trained  technicians  using  conventional  procedures. 

A  second  study  compared  high  school  students,  using  Job  aids,  to  highly 
trained  electronics  technicians,  using  standard  manuals.  In  that  study, 
high  school  students  were  given  12  hours  of  training  on  Identification  of 
electronic  components,  using  a  volt  meter,  and  basic  soldering  procedures. 
The  technicians,  on  the  other  hand,  had  considerable  formal  training  In 
electronics,  the  aiajorlty  having  three  to  six  years  of  field  experience. 

Both  groups  were  assigned  to  tasks  Involving  complex  awlntenance  of 
electronic  equipment.  The  students  using  the  Job  aids  outperformed  the 
technicians  In  every  phase  of  the  study.  One  subject  In  that  experiment,  in 
the  skilled  technician  group,  required  12  hours  to  troubleshoot  a  radar 
system,  performed  133  steps,  and  referred  to  41  different  sections  of  8 

separate  manuals.  A  high  school  student,  using  the  Job  aid,  required  five 

hours  to  perform  35  steps,  and  referred  to  only  three  different  manuals. 

The  bottom  line  Is  that  a  fair  amount  of  data  exists  to  testify  to  the 
potential  benefits  of  are  1 1 -designed  Job  aids  In  reducing  the  need  for 
extensive  training  on  new  tasks. 

Jack  Foil ey.  In  1975,  addressed  the  Issue  of  comparing  performance 
based  exclusively  on  training  to  that  based  exclusively  upon  Job  aids.  He 

hypotheslxed  that  If  we  allocate  the  entire  requirement  to  training,  we 

might  achieve  a  relatively  low  level  of  performance,  because  of  the  large 
amount  of  Information  required  to  be  learned  In  a  complex  skill  area. 
Similarly,  If  one  allocates  the  requirement  entirely  to  Job  aids,  we  also 
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■ay  get  a  relatively  low  level  of  perforaance,  due  to  the  fact  that  the 
performer  aay  become  tied  to  the  aid.  If  the  aid  contains  an  error,  the 
performer  gets  lost.  Folley  has  come  up  with  a  gradient  that  describes  the 
tradeoffs  between  these  two,  and  generates  a  hypothetical  opt  leal  mixture  of 
these  two  functions. 

The  Navy's  EPICS  program  uses  an  alternative  approach,  which  also 
Includes  the  use  of  job  aids.  It  provides  for  ao-called  ‘enriched"  and 
"hybrid"  Job  aids.  A  "hybrid"  aid  la  one  that  presents  troubleshooting 
Information,  both  in  deductive  and  directive  formats,  whereas  an  "enriched" 
aid  Is  one  that  presents  additional  Job  Inforaiatlon  to  facilitate  the 
transition  between  directive  and  deductive  formate. 

Because  of  Job  aids,  the  definition  of  transfer  may  need  modification. 
In  some  sense,  the  process  of  transfer  implies  at  least  two  phases.  The 
first  Is  when  general  Information  Is  recovered  from  prior  experience  for 
transfer  to  a  new  situation.  The  second  Is  the  direct  application  of  that 
prior  experience  to  current  performance  or  to  a  current  learning  situation. 

A  Job  aid  Is  often  used  as  an  aid  to  memory,  by  not  requiring  a  person 
to  recover  something  from  past  experience  to  apply  to  the  current  situation. 
It  unburdens  an  individual's  memory,  and  therefore  facilitates  transfer 
which  otherwise,  might  not  be  possible.  The  point  le,  should  "transfer”  be 
defined  only  in  terms  of  having  to  tap  one's  awmory  to  retrieve  Information? 

I  suggest  that  Improved  technical  documentation.  Including  Job  aids,  is 
an  area  where  major  strides  can  be  made  In  DoD.  Certainly,  the  demonstrated 
effectiveness  of  artificial  Intelligence-based  diagnostic  Information 
systems  le  one  major  area  where  Improvement  has  been  made.  The  use  of 
handheld  tutors  In  the  on-the-job  environment  le  another,  yet  much  more  work 
remains  to  be  accomplished  In  these  areas.  I  think  that  Improved  technical 
documentation  can  reduce  many  of  DoD's  transfer  of  training  problems. 

I  have  commented  In  five  areas  of  training  and  transfer  that  I  think 
are  Important  to  those  of  us  who  are  In  the  research  community.  I  wish  to 
emphasize  that  these  and  other  similar  areas  of  Interest  must  be  developed 
and  tried  out  In  applied  settings  before  any  meaningful  conclusions  can  be 
drawn. 
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SIMAARY  RECOMMENDATIONS 


Following  the  presentations,  all  participants  summarized  main  points 
from  the  workshop  and  made  suggestions  for  future  research  and  related 
activities.  Following  are  the  primary  recommendations  raised  during  the 
summary  discussion.  They  are  the  recommendations  of  one  or  several 
Individuals,  and  do  not  necessarily  reflect  a  consensus  of  opinion.  The 
recommendations  have  been  combined  by  specific  area. 


ISSUE 


RECOMMENDATION 


Definitions 


Interventions 


Measures 


Boaaarch  Hoads 


There  Is  a  need  for  standard,  operational 
definitions  of  several  terms.  Including  "transfer 
of  training."  "far  transfer,"  and  “near  transfer." 

For  the  benefit  of  researchers  who  are  uninformed 
of  past  attempts  at  Interventions,  a  listing  should 
be  prepared  of  Interventions  that  have  already 
succeeded  (or  failed)  In  a  military  setting. 

A  listing  of  needed  Interventions  should  be 
developed,  as  well. 

The  research  community  should  develop  a  listing  of 
■wasures  of  transfer.  Including  the  data  that  are 
needed  In  order  to  calculate  them.  When  measures 
are  not  currently  available,  the  research  community 
should  determine  what  methods  are  needed  to  develop 
them.  This  would  Include  measures  for  both  actual 
Job  performance  and  prediction  of  Job  performance. 

There  Is  a  need  for  Investigation  of  team 
performance,  the  necessary  mixes  of  skills  on 
teams,  and  a  team  person/ Job  match  optimization 
system. 

The  relationship  between  Job  performance  and 
training  conditions  needs  to  be  Investigated 
further,  and  should  Include  a  substantial  data 
col  lection  effort. 


There  I*  a  need  for  a  theoretical  base  for  the 
development  of  unpracticed  elcllla. 


Hlgh-f Idel Ity,  low-coat  performance  measures  need 
to  be  developed  to  aaaeas  transfer  of  training  and 
to  validate  predictor  testa. 


In  developing  Intelligent  job  aids  and  other 
Instructional  devices,  researchers  should 
anticipate  the  transfer  situation. 

Systematic  instructional  development  processes, 
Including  feedback  and  monitoring  systems,  need 
Improvement . 

The  training  community  should  make  known  the  types 
of  tools  that  are  needed,  and  the  research 
community  should  work  on  the  algorithms  for  the 
development  of  tools. 

Researchers  should  become  knowledgeable  about  the 
content  matter  of  the  training  task  (e.g., 
electronics)  before  attempting  to  devise  training 
programs. 

Da  1 1  very  of  Training  There  should  be  development  of  Job  aids  (possibly 

computer-based)  aimed  at  the  changing  expertise  of 
the  trainee. 

The  Implementation  of  training  programs  should  be 
aionltored  more  effectively. 

Training  environments  should  be  structured  to  allow 
for  sufficient  training  trials,  to  promote  higher 
levels  of  skill  development. 

Add  I icat ions  Researchers  should  be  more  cognizant  of  the 

potential  applications  of  their  work.  What 
applications  do  the  participants  anticipate  from 
their  work? 

Academic-based  researchers  need  encouragement, 
possibly  through  a  funding  Initiative,  to  produce 
more  research  with  direct  tppl Icat Ion  to  transfer 

Issues. 


Das Ion  of 
Instruction 


70 


funding 


Workplace/Eaulpment 
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Task.  Analysis 


Future  Discussions 


The  workplace  environment  and  equipment  design  must 
be  considered  when  developing  job  aids  and  related 

devices. 


Psychologists  should  communicate  research  results 
to  the  user  community  using  terminology  appropriate 
for  that  group. 

Basic  and  applied  researchers  should  communicate 
more  effectively  with  one  another  to  stimulate  work 
In  both  direct  Ions. 

Analysis  of  tasks  should  be  based  on  underlying 
cognitive  and  psychomotor  processes  needed  for 
performance  of  specific  functions. 

Task  analysis  Is  needed  to  specify  the  cognitive 
processes  and  memory  structures  In  performance 
constructs. 

There  should  be  more  opportunities  for  scientists 
from  the  military  laboratories,  and  both  academic 
and  non-academic  settings  to  discuss  current 
research  and  future  research  needs,  it  was  also 
recommended  that  representatives  of  other 
professions  (e.g.,  Job  aid  development)  would 
provide  useful  comments. 
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